One year of ResNet Gitlab

Today, it has been one year since the first Merge Request (MR) was created¬†and accepted by ResNet* Gitlab. During that time, about 250 working days, we have processed 462 MRs as part of our Puppet workflow. That’s almost two a day!

We introduced Git and Gitlab into our workflow to replace the ageing svn component which didn’t handle branching and merging well at all. Jumping to Git’s versatile branching model and more recently adding r10k into the mix has made it trivially easy to spin up ephemeral dev environments to work on features and fixes, and then to test and release them into the production environment safely.

We honestly can’t work out how on earth we used to cope without such a cool workflow.

Happy Birthday, ResNet Gitlab!

* 1990s ResNet brand for historical reasons only – this Gitlab installation is used mostly for managing eduroam and DNS. Maybe NetOps would have been a better name ūüôā

Integrating Puppet and Git

Introduction

One of the main benefits of working with Git is the ability to split code into branches and work on those branches in parallel. By integrating your Git branches into your Puppet workflow, you can present each Git branch as a Puppet environment to separate your dev, test and prod environments in different code branches.

This guide assumes you already have a working Puppet master which uses directory environments, and a working Gitlab install. It assumes that your Puppet and Gitlab instances are on different servers. It’s recommended that you install Gitlab with Puppet as I described in a previous blog post.

Overview

The basic principle is to sync all Puppet code from Gitlab to Puppet, but it’s a little more complex than this. When code is pushed to any branch in Git, a post-receive hook is executed on the Gitlab server. This connects via ssh to the Puppet master and executes another script which checks out the code from the specific Git branch into a directory environment of the same name.

Deploy keys

First you need to create a deploy key for your Puppet Git repo. This grants the Puppet master read-only access to your code. Follow these instructions to create a deploy key. You’ll need to put the deploy key in¬†the home directory of the puppet user on your Puppet master, which is usually /var/lib/puppet/.ssh/id_rsa.pub.

Puppet-sync

The puppet-sync script is the component on the Puppet master. This is available from Github and can be installed verbatim into any path on your system. I chose /usr/local/bin.

SSH keys

You’ll need to create¬†a pair of SSH public/private keys to allow your Gitlab server to ssh to your Puppet server. This is a pretty standard thing to do, but instructions are available on the README for puppet-sync.

Post-receive hook

The post-receive hook must be installed on the Gitlab server. The source code is available from Github. You will need to modify the hook to point at the right locations. The following values are sensible defaults.

REPO="git@gitlab.yoursite.com:namespace/puppet.git"
DEPLOY="/etc/puppet/environments"
SSH_ARGS="-i /var/opt/gitlab/.ssh/git_id_rsa"
PUPPETMASTER="puppet@puppet.yoursite.com"
SYNC_COMMAND="/usr/local/bin/puppet-sync"

If you are using the spuder/gitlab Puppet module to manage your Gitlab instance, installing a custom hook is as simple as doing:

gitlab::custom_hook { 'puppet-post-receive':
  namespace => 'namespace',
  project   => 'puppet',
  type      => 'post-receive',
  source    => 'puppet:///modules/site_gitlab/puppet-post-receive',
}

If you installed Gitlab by hand, you’ll need to do manually install the hook into the hooks subdirectory in your raw Git repo. On our Gitlab installation, this is in¬†/var/opt/gitlab/git-data/repositories/group/puppet.git/hooks

Workflow

Now all the components are in place, you¬†can start to use it (when you’ve tested it). Gitlab’s default branch is called master whereas Puppet’s default environment is production. You need to create a new branch in Gitlab¬†called production, and set this to be the default branch. Delete master.

  1. Make new feature branches on the Gitlab server using the UI
  2. Use git pull and git checkout to fetch these branches onto your working copy
  3. Make changes, commit them, and when you push them, those changes are synced to a Puppet environment with the same name as the branch
  4. Move a dev VM into the new environment by doing puppet agent -t --environment newfeature
  5. When you’re happy, send a merge request to¬†merge your branch back into production. When the merge request is accepted, the code will automatically be pushed to the Puppet server.

Read about this in more detail: Gitlab Flow.

soc::puppet – A puppet themed social event for UoB (Thursday 19th March)

What: soc::puppet is a puppet themed meet up for University of Bristol Staff using, or interested in puppet configuration management  (rather than actual marionettes or glove puppets)
Where: Brambles in The Hawthorns (see the link for details)
When: 5pm-7pm(ish) Thursday 19th March 2015

There’s a growing¬†community of¬†people around the University of Bristol using (or interested in using)¬†puppet¬†configuration management http://puppetlabs.com Some of those people are talking to eachother, and some just don’t know who to talk to yet!

Experience, use case and scale of implementation varies widely, but we’ve all got something to share! ūüôā

With that in mind, there seems to be interest in an informal gathering of interested people, where we can get together, share ideas and build a local puppet community.  Bringing together all those informal corridor/tearoom chats and spreading the exciting ideas/knowledge around in a loose, informal manner.

As a first pass, we’ve booked “Brambles” which is the new name for the Staff Club space in The Hawthorns, for a couple of hours after work on¬†Thursday 19th March. ¬†If it goes well, it will hopefully turn into a regular event.

Our initial aim is to make it as informal as possible (hence doing it outside work hours, no pressure to minute it, assign actions, instigate formalised project teams etc) and treat it mostly as an exercise in putting people in touch with other people who are playing with similar toys.

That said, there are a few “bits of business” to take care of at the first meeting, so I’m suggesting the following as a vague agenda.

  1. Welcome! ¬†What’s this about? (about 5 minutes)
  2. Introductions, very quick “round table” to introduce¬†everyone, and say what level of exposure they’ve had to puppet so far (about 10 minutes)
  3. Everything beyond this point will be decided on the day. ¬†If you’ve got something you’d like to talk about or present on, bring it with you!
  4. We’ll close the session with a very quick “should we do this again?” and “call for volunteers”

If people are interested, we can move on to a pub afterwards to continue the discussion.

The facilities available are a bit limited,¬†and apparently the projector isn’t¬†available at the moment, but we’ll see what direction it takes – and as they say in Open Space circles, “Whatever happens is the only thing that could have, be prepared to be surprised!”

Puppet future parser — what to expect that you’ll have to update in your manifests…

The Puppet Future Parser is the new implementation of the manifest parser which will become the default in 4.0, so I thought I’d take a look to see what I’d need to update.

Also, there are some fancy new features like iteration and that you can use [1,2] array notation or {a=>b} hash notation anywhere that you’d previously used a variable containing an array or hash.

The iteration and lambda features are intended to replace create_resources calls, as they are more flexible and can loop round repeatedly to create individual definitions.

For example, here’s a dumb “sudo” profile which uses the each construct to iterate over an array:

class profiles::sudo {
  # This is a particularly dumb version of use of sudo, to allow any commands:
  $admin_users = hiera_array('admin_users')
  # Additional users with special sudo rights, but no ssh access (e.g. root):
  $sudo_users  = hiera_array('sudo_users')

  class { ::sudo: }

  $all_sudo_users = concat($sudo_users, $admin_users)

  # Create a resource for each entry in the array:
  each($all_sudo_users) |$u| {
    sudo::entry { $u:
      comment  => "Allow ${u} to run anything as any user",
      username => $u,
      host     => 'ALL',
      as_user  => 'ALL',
      as_group => 'ALL',
      nopasswd => false,
      cmd      => 'ALL',
    }
  }
}

Making this work with create_resources and trying to splice in the the username for each user in the list into a hash looked like it would be messy, requiring at least an additional layer of define — this method is much neater.

This makes it much easier to create data abstractions over existing modules — you can programmatically massage the data you read from your hiera files and call definitions using that data in a much more flexible way than when passing hashes to create_resources. This “glue” can be separated into your roles and profiles (which could be the subject of another post but are described well in this blog post), creating a layer which separates the use of the module from the data which drives that use nicely.

So this all sounds pretty great, but there are a few changes you’ll possibly encounter when switching to the future parser:

  • Similar to the switch from puppet master to puppet server, the future parser is somewhat more strict about data formats. e.g. I found that my hiera data definitely needed to be properly quoted when I started using puppet server, so entries like mode : 644 in a file hash wouldn’t give the number you were expecting… (needs mode : 0644 or mode : '644' to avoid conversion from octal to decimal…). The future parser extends this to being more strict in your manifests, so a similarly-incorrect file { ... mode => 644 } declaration needs quoting or a leading zero. If you use puppet-lint you’ll catch this anyway — so use it! ūüôā
  • It’s necessary to use {} instead of undef when setting default values for hiera_hash (and likewise [] instead of undef for hiera_array), to allow conditional expressions of the form if $var { ... } to work as intended. It seems that in terms of falseness for arrays and hashes that undef is in fact true… (could be a bug, as this page in the docs says: “When used as a boolean, undef is false”)
  • Dynamically-scoped variables (which are pretty mad and difficult to follow anyway, which is why most languages avoid them like the plague…) don’t pass between a class and any sub-classes which it creates. This is in the docs here, but it’s such a common pattern that it could well have made it through from your old (pre-Puppet 2.7) manifests and still have been working OK until the switch to the future parser. e.g.:
    class foo {
      $var = "x"
    }
    
    class bar {
      include foo
      # $var isn't defined here, as dynamic scope rules don't allow it in Puppet >2.7
    }
    

    Instead you need to explicitly qualify your variables to pull them out of the correct scope — $foo::var in this case. In your erb templates, as a common place where the dynamically-scoped variables might have ended up getting used, you can now use scope['::foo::var'] as a shorthand for the previously-longer scope.lookupvar('::foo::var') to explicitly qualify the lookup of variables. The actual scope rules for Puppet < 2.7 are somewhat more complicated and often led to confusing situations if you unintentionally used dynamic scoping, especially when combined with overriding variables from the parent scope…

  • I’m not sure that expressions of the form if "foo" in $arrayvar { ... } work how they should, but I’ve not had a chance to investigate this properly yet.

Most of these are technically the parser more strictly adhering to the specifications, but it’s easy to have accidentally had them creep into your manifests if you’re not being good and using puppet-lint and other tools to check them.

In conclusion : Start using the Future Parser soon! It adds excellent features for iteration which make abstracting data a whole lot easier than using the non-future (past?) parser allows. Suddenly the combination of roles, profiles and the iteration facilities in the future parser mean that abstraction using Puppet and hiera makes an awful lot more sense!

Using Puppet to deploy code from Git

I’ve¬†revisited the way that we at ResNet deploy our web applications to web servers. We decided to store the application code in a Git repository. As part of our release process, we create a tag in Gitlab.

Rather than check the code out manually, we are using a Forge module called puppetlabs/vcsrepo to clone a tagged release and deploy it. Our app repos do not permit anonymous cloning so the Puppet deployment mechanism must be able to authenticate. I found the documentation for puppetlabs/vcsrepo to be a bit lacking and had spend a while figuring out what to do to make it work properly.

I recommend you generate a separate SSH key for each app you want to deploy. I generated my key with ssh-keygen and added it to Gitlab as a deploy key which has read-only access to the repo – no need to make a phantom user.

Here’s a worked example with some extra detail about how to deploy an app from git:

# Define docroot
$docroot = '/var/www/app'

# Deploy SSH key to authenticate git
file { '/etc/pki/id_rsa':
  source => 'puppet:///modules/app/id_rsa',
  owner  => 'root',
  group  => 'root',
  mode   => '0600',
}
file { '/etc/pki/id_rsa.pub':
  source => 'puppet:///modules/app/id_rsa.pub',
  owner  => 'root',
  group  => 'root',
  mode   => '0644',
}

# Clone the app from git
vcsrepo { 'app':
  ensure   => present,
  path     => $docparent,
  provider => git,
  source   => 'git@gitlab.resnet.bris.ac.uk:resnet/app.git',
  identity => '/etc/pki/git_id_rsa',
  revision => '14.0.01',
  owner    => 'apache', # User the local clone will be created as
  group    => 'apache',
  require  => File['/etc/pki/id_rsa', '/etc/pki/id_rsa.pub'],
}

# Configure Apache vhost
apache::vhost { 'app'
  servername    => 'app.resnet.bris.ac.uk',
  docroot       => $docroot,
  require       => Vcsrepo['app'],
  docroot_owner => 'apache',  # Must be the same as 'owner' above
  docroot_group => 'apache',
  ...
}

To deploy a new version of the app, you just need to create a new tagged release of the app in Git and update the revision parameter in your Puppet code. This also gives you easy rollback if you deploy a broken version of your app. But you’d never do that, right? ūüėČ

Publish a Module on Puppet Forge

I’ve started publishing as many of my Puppet modules as possible on Puppet Forge. It isn’t hard to do but there are a few things to know. This guide is largely based on Puppetlabs’ own guide Publishing Modules on the Puppet Forge.

  1. For home-grown modules that have grown organically, you are likely to have at least some site-specific data mixed in with the code.¬†Before publishing, you’ll need to abstract this out. I recommend using parametrised classes with sane defaults for your inputs. If necessary, you can have a local wrapper class to pass site-specific values into your module.
  2. The vast majority of Puppet modules are on GitHub, but this isn’t actually a requirement. GitHub offers public collaboration and issue tracking, but you can keep your code wherever you like.
  3. Before you can publish, you need to include some metadata with your module. Look at the output of puppet module¬†generate. If you’re starting from scratch, this command is an excellent place to start. If you’re patching up an old module for publication, run it in a different location and selectively copy the useful files into your module. The mandatory files are metadata.json and README.md.
  4. When you’re ready to publish, run puppet module build. This creates a tarball of your module and metadata which is ready to upload to Puppet Forge.
  5. Create an account on Puppet Forge and upload your tarball. It will automatically fill in the metadata.
  6. Install your module on your Puppetmaster by doing puppet module install myname/mymodule

Building a Gitlab server with Puppet

GitHub is an excellent tool for code-sharing, but it has the major disadvantage of being fully public. You probably don’t want to put your confidential stuff and shared secrets in there! You can pay for¬†private repositories, but the issue still stands that we shouldn’t be putting confidential UoB things in a non-approved cloud provider.

I briefly investigated several self-hosted pointy-clicky Git interfaces, including Gitorious, Gitolite, GitLab, Phabricator and Stash. They all have their relative merits but they all seem to be a total pain to install and run in a production environment, often requiring that we randomly git clone something into the webroot and then not providing a sane upgrade mechanism. Many of them have dependencies on modules not included with the enterprise Linux distributions

In the end, the easiest-to-deploy¬†option seemed to be to use the GitLab Omnibus installer. This bundles the GitLab application with all its dependencies in a single RPM for ease of deployment. There’s also a Puppet Forge module called spuder/gitlab which makes it nice and easy to install on a Puppet-managed node.

After fiddling, my final solution invokes the Forge module like this:

class { 'gitlab' : 
  puppet_manage_config          => true,
  puppet_manage_backups         => true,
  puppet_manage_packages        => false,
  gitlab_branch                 => '7.4.3',
  external_url                  => "https://${::fqdn}",
  ssl_certificate               => '/etc/gitlab/ssl/gitlab.crt',
  ssl_certificate_key           => '/etc/gitlab/ssl/gitlab.key',
  redirect_http_to_https        => true,
  backup_keep_time              => 5184000, # 5184000 = 60 days
  gitlab_default_projects_limit => 100,
  gitlab_download_link          => 'https://downloads-packages.s3.amazonaws.com/centos-6.5/gitlab-7.4.3_omnibus.5.1.0.ci-1.el6.x86_64.rpm',
  gitlab_email_from             => 'gitlab@example.com',
  ldap_enabled                  => true,
  ldap_host                     => 'ldap.example.com',
  ldap_base                     => 'CN=Users,DC=example,DC=com',
  ldap_port                     => '636',
  ldap_uid                      => 'uid',
  ldap_method                   => 'ssl',
  ldap_bind_dn                  => 'uid=ldapuser,ou=system,dc=example,dc=com',
  ldap_password                 => '*********',
}

I also added a couple of resources to install the certificates and create a firewall exception, to make a complete working deployment.

The upgrade path requires manual intervention, but is mostly automatic. You just need to change gitlab_download_link to point to a newer RPM and change gitlab_branch to match.

If anyone is interested, I’d be happy to write something about the experience of using GitLab¬†after a while, when I’ve found out some of the quirks.

Update by DaveG! (in lieu of comments currently on this site)

Gitlab have changed their install process to require use of their repo, so this module doesn’t like it very much. They’ve also changed the package name to ‘gitlab-ce’ rather than just ‘gitlab’.

To work around this I needed to:

  • Add name => 'gitlab-ce' to the package { 'gitlab': ... } params in gitlab/manifests/install.pp
  • Find the package RPM for a new shiny version of Gitlab. 7.11.4 in this case, via https://packages.gitlab.com/gitlab/gitlab-ce?filter=rpms
  • Copy the RPM to a local web-accessible location as a mirror, and use this as the location for the gitlab_download_link class parameter

This seems to have allowed it to work fine!
(Caveat: I had some strange behaviour with whether it would run the gitlab instance correctly, but I’m not sure if that’s because of left-overs from a previous install attempt. Needs more testing!)

Puppet — making array expansions in resource calls unique

Lets say that you have an array of NFS clients to allow access to an export. You want to expand this into a list of resources, which you’d normally do via something like:

$allow = hiera_array('some_variable', undef)
nfs::export { $allow:
  ...
}

This works fine until you want to have multiple exports to the same name/IP, at which point you have two nfs::export resources with the same name. But you need the expansion of the array to generate multiple calls to nfs::export, so you need to prefix each string in the array:

$allow = hiera_array('some_variable', undef)
$hacked_allow = prefix($allow, "${fqdn} export of ${path} to client ")
nfs::export { $hacked_allow:
  ...
}

..then in the called define you extract the client fqdn like so:

define nfs::export ( ... ) {
  $client = regsubst($name, "^.* export of .* to client (.*)$", '\1')
  validate_string($client)

  ... [use $client as fqdn in here]
}

This seems like a terrible hack, but it’s a reasonable workaround for the need to expand arrays and keep the names unique to avoid duplicate resources.

mcollective with SSL — the short(ish) version

Prerequisites

  • A working puppet master and some clients to test with
  • Hiera in use, with some “common” section which is applied to all nodes. This isn’t strictly required, but it makes the set-up a lot simpler.

What you’ll end up with

  • SSL’d connections between all involved daemons
  • An admin user who can run “mco ping” and get responses from all the mcollective-enabled nodes

Basic setup (without SSL)

First you’ll want to get mcollective up and running *without* SSL to be sure that the module is working, before leaping on to use SSL and certificates.

Since this is a very basic setup all the examples with use the puppet master as the “broker”, where activemq queues all the requests and replies centrally, rather than a separate machine. Replace the puppet.example.org and node[12].example.org references with your host names as appropriate ūüôā

  • Install the puppetlabs mcollective module and its dependencies with:
    puppet module install puppetlabs-mcollective
    This installs many other modules to help it along.
  • Add the mcollective class to your included class list for all nodes. e.g.:
    classes:
      ... (probably plenty of others)
      - mcollective
    
  • Add these variables to your hiera configs for all hosts:
    mcollective::middleware_hosts: puppet.example.org
    
  • Add these variables to apply only to your puppet master:
    mcollective::middleware: true
    mcollective::server    : true
    mcollective::client    : true
    
  • If you have firewalling on your test puppet master (and you should!), open port 61613. For me this required adding another hiera entry:
     site::firewall:
       '040 accept all stomp/activemq connections':
         state : ['NEW']
         proto : 'tcp'
         dport : 61613
         action: accept
    
  • Now run puppet (or wait for a run to occur automatically, if you have that set up) on the puppet master and a couple of nodes, and you should get the mcollective agent installed and configured on all nodes, plus the activemq daemon set up and running on the broker.

At this point you should be able to run mco ping and get some ping time results back from the nodes which are running the mcollective agent. You can also look at the output of netstat on the activemq server and see a number of ESTABLISHED connections to the 61613 port from the mcollective nodes.

Adding SSL support

Now you have a basic set-up working the steps required to get SSL working are :

  • Allow firewall connections to port 61614 on the broker (same host as the puppet master in this simple case)
  • Generate a strong random password for the mcollective user — I use pwgen -s -y 20 to generate good long random passwords
  • Generate a certificate (public/private) pair for the new admin user, using puppet cert generate admin-user-name on the puppet master
  • Generate a certificate (public/private) pair for the mcollective and activemq daemons to use, with puppet cert generate mcollective-servers
  • Create a site-local mcollective certificates set of directories under your module directories. For example, we have:
      /etc/puppet/modules/project_zoned/files/mcollective/
      /etc/puppet/modules/project_zoned/files/mcollective/client_certs
      /etc/puppet/modules/project_zoned/files/mcollective/certs
      /etc/puppet/modules/project_zoned/files/mcollective/private_keys
    
  • Copy the certificates to the appropriate places:
      cp /var/lib/puppet/ssl/certs/ca.pem /etc/puppet/modules/project_zoned/files/mcollective/certs/
      cp /var/lib/puppet/ssl/certs/mcollective-servers.pem /etc/puppet/modules/project_zoned/files/mcollective/certs/
      cp /var/lib/puppet/ssl/private_keys/mcollective-servers.pem /etc/puppet/modules/project_zoned/files/mcollective/private_keys/
      cp /var/lib/puppet/ssl/certs/admin-user-name.pem /etc/puppet/modules/project_zoned/files/mcollective/client_certs/
      cp /var/lib/puppet/ssl/private_keys/admin-user-name.pem /etc/puppet/modules/project_zoned/files/mcollective/private_keys/
      cp /var/lib/puppet/ssl/certs/admin-user-name.pem /etc/puppet/modules/project_zoned/files/mcollective/client_certs/
    
  • Add a declaration of an mcollective::user. We have various site-local classes which get auto-expanded into multiple calls to puppet defines, so we add a list like this:
    site::mcollective::users:
      - 'admin-user-name':
        group : 'admin-user-group'
        certificate: 'puppet:///modules/project_zoned/mcollective/client_certs/admin-user-name.pem'
        private_key: 'puppet:///modules/project_zoned/mcollective/private_keys/admin-user-name.pem'
    
  • Re-run puppet on the puppet master and you’ll have an install of mcollective which is set up to use SSL, and an admin user who has a ~/.mcollective config file, along with keys in their ~/.mcollective.d directory. This user should be able to run mco ping and get results from any nodes which have had the mcollective setup installed on them too (which is any that you run puppet on after adding the mcollective class and settings into a common yaml file for all nodes).

Now you should have a basic SSL-encrypted setup which works, and you can start adding mcollective plugins to do useful stuff like manage services and the puppet agent runs etc.

Adding more admin users just requires generating certificates for the user, copying them to the mcollective files repository in your “project” module, and adding them to the site::mcollective::users hash. Then they get the config added to their homedir automatically.

I can see this being rather useful as the number of machines we are managing continues to grow ūüėÄ

Docs which were useful / further reading