Puppet tutorial series

• Part 1: Powering up with Puppet
• Part 2: Client and Server

In Part 1 of this Puppet tutorial we saw how to install Puppet on a Linux machine from source, and create a basic manifest which controls the NTP service. In this episode we’ll cover setting up a Puppet server and then using it to control multiple client machines.

Install Puppet from packages

In the first part of the tutorial, we installed Puppet directly from the source. This is a great way to learn and experiment, but for production purposes we would like to use a standard package - in this case, an RPM package we can install via Yum or Cobbler. This way, you can ensure that the same version of Puppet is present on your servers and update it automatically if need be. The RPM also has a few bonus features that the source package doesn’t.

If you want to follow along with the tutorial using your existing Puppet source install, that’s fine. It’s a good idea to make sure you’re using the same version of Puppet on both client and server, or you may run into obscure and annoying problems, so either install from the same source package on your client machine, or set up two fresh machines with the RPM package as described below.

If you’re using RHEL or CentOS then the EPEL repository is the best place to start. Enable the repository by installing the correct package for your OS version, following the instructions on How to Use EPEL:

# rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/x86_64/epel-release-5-3.noa...
Retrieving http://download.fedora.redhat.com/pub/epel/5/x86_64/epel-release-5-3.noa...
warning: /var/tmp/rpm-xfer.2O73tR: Header V3 DSA signature: NOKEY, key ID 217521f6
Preparing...                ########################################### [100%]
   1:epel-release           ########################################### [100%]

You can then run:

# yum install puppet-server

and the Puppet server package will be installed from EPEL along with a bunch of dependencies. (Note that in EPEL puppet and puppet-server are two separate packages; on the client machines you’ll only need to install puppet.)

You can now start the Puppetmaster daemon and make the service persistent:

# chkconfig puppetmaster on
# service puppetmaster start
Starting puppetmaster:                                     [  OK  ]

If there is a firewall between the Puppetmaster and its clients (eg an iptables firewall on the Puppetmaster server itself) you will need to open TCP port 8140 to the clients.

Create a client manifest

In Part 1 of the Puppet Tutorial we created a file /etc/puppet/manifests/nodes.pp, which lists the nodes (machines) that Puppet knows about, and what configuration they should have. Currently it looks like this:

node myserver {
    include ntp
}

As we’re adding a new node to the system, we need to modify the file so that it reads like this:

node myserver {
    include ntp
}

node myclient {
    include ntp
}

Authorising a client

Puppet uses SSL (Secure Sockets Layer), an encrypted protocol, to communicate between master and clients. This means that only a client with a correctly signed SSL certificate can access the Puppetmaster and receive its configuration. To exchange certificates between the master and client, follow this procedure.

Adding a ‘puppet’ host entry

By default Puppet will look for a host called puppet on the local domain. To make this work, add the IP address of the Puppetmaster into the /etc/hosts file on the client, like this:

83.222.113.61   puppet

Test this with ping puppet (assuming you have allowed ICMP in the firewall).

Generate a certificate request

On the client, run:

# puppetd --test
info: Creating a new certificate request for myclient.mydomain.com
info: Creating a new SSL key at /etc/puppet/ssl/private_keys/myclient.mydomain.com.pem
warning: peer certificate won't be verified in this SSL session
notice: Did not receive certificate
notice: Set to run 'one time'; exiting with no certificate

Sign the certificate

On the master:

# puppetca -l
myclient.mydomain.com

There is a certificate request pending for myclient. To sign it, run:

# puppetca -s myclient.mydomain.com
notice: Signed certificate request for myclient.mydomain.com
notice: Removing file Puppet::SSL::CertificateRequest myclient.mydomain.com at '/var/lib/puppet/ssl/ca/requests/myclient.mydomain.com.pem'

If there is no certificate request listed, the client was not able to contact the server for some reason. Check that you have the right IP address for puppet in /etc/hosts on the client, and that TCP port 8140 is open on both the master and client firewalls. The puppetmasterd daemon also needs to be running on the master.

Run Puppet for the first time

On the client, you should now be able to run:

# puppetd --test
notice: Got signed certificate
info: Caching catalog at /var/puppet/state/localconfig.yaml
notice: Starting catalog run
notice: //ntp/Service[ntpd]/ensure: ensure changed 'stopped' to 'running'
notice: Finished catalog run in 0.92 seconds

Just as on the server in the first part of the tutorial, Puppet is now applying the ntp class to myclient. At the moment all this class does is ensure the ntp service is running, but of course it could do many more interesting things. And that’s what we’ll look at next time in Part 3.

The problem

Is this a familiar nightmare? “The server you built yesterday is great,” your boss enthuses. “Now we need a hundred more just like it. And when you’ve done that, could you take a look at my laptop before lunch? I can’t seem to get into my email.”

It’s common nowadays for sysadmins to manage very large numbers of servers - often virtual servers created on-demand in cloud platforms like Amazon EC2. How do you build so many servers and keep them updated - automatically, reliably, and quickly?

One common strategy is the golden image pattern: set up the server the way you want it, and then copy the disk image and use that to boot as many clone machines as you need. Another is the automation pattern: start with a generic server, and then apply automatic configuration tools such as Puppet to bring the machine into a specified state where it is ready for production.

The golden image

The golden image is attractive to many because it’s easy and cheap. It’s quick to create images, and you just need a big disk to store them.

Imaged machines also have low latency: they’re ready for production in just a few minutes from startup. Finally, it’s platform agnostic: you can image virtually any kind of machine, even Windows (which most config automation tools can’t handle very well).

So what’s the problem with the golden image? Luke Kanies, creator of Puppet, prefers to call it the “foil ball” approach: your images can quickly become an unmanageable, crumpled mess. Firstly, how many images do you need? Well, as many as you have different types of machines: web servers, databases, load balancers, DNS servers, and so on.

The number of images multiplies again when you take history into account. What happens when you need to recreate the state of the database server last month to troubleshoot a problem? You need to keep old images around, which means a new image every time you make a change.

And here’s a thing: if you do need to make a change (for example a security upgrade for a critical package), you’ll have to rebuild the golden image and then re-image and reboot all affected machines - which means lots of bandwidth and possibly downtime. And, unfortunately, even imaged machines usually need some final tweaking (eg IP addresses) before they’re ready for use.

Automation tools

By comparison, building your machines with an automatic configuration tool such as Puppet, Chef or even cfengine gives you many advantages. The configuration is self-documenting; it’s easy to see what should be on each machine, and more importantly, why it’s there. It will also stay in sync with reality.

Although your server types will be different, they will also likely have a lot in common. With images, these supposedly common settings could easily drift apart over time. With automation tools, you can put these settings into modules or classes, just as a software developer re-uses a module or library in another project.

In fact, all the advantages of software modules are available: you can share code with others, download public recipes and manifests, and maintain a complete version history of your network.

Automated configuration is intrinsically multi-platform. Puppet and Chef can abstract away some of the differences between operating systems, and you can add minor tweaks of your own. For example, you might need a slightly different MySQL config setting for MySQL 4.x as opposed to 5.x. A one-line switch in the configuration can address this problem, instead of creating two disk images.

Of course, everything comes with a downside. It is more difficult and usually more time-consuming to build an automated configuration, especially from scratch and when you’re still learning to use the tools. And some things are hard to automate. Some software just wasn’t written with automated installation and management in mind.

Also, not all software will be packaged for your platform (or you need special compilation options). In these cases you will need to package the software yourself.

Finally, configuration management doesn’t always mix well with human intervention. In particular, someone may make a config change on the machine which is then automatically reversed by Puppet. At best this can be a nasty surprise, at worst it can result in a site outage. Managing desktops automatically can create similar problems. This needs to be addressed at a business level with a proper change control process and making it clear to everyone where the boundaries of automatic management lie.

Combined strategies

So both imaging and automation have their place; neither is a magic bullet. The best strategy for scaling your deployments is often a judicious combination of these two approaches. You can, for example, start with a ‘stem cell’ image containing an approved base build, security fixes and settings which don’t often change, which is then configured by Puppet or Chef. Or you could use provisioning tools such as Kickstart and Cobbler and then hand over to Puppet. Or even build the machine with Puppet and then create a golden image from this.

One thing is certain: ignoring the problem is not an option. Moving to an automated network might seem like a big project, but the key is to start small. Set aside a block of time every week to spend on improving your infrastructure, and automate one small thing at a time as part of a gradual continuous process. The more you invest in this, the more benefits you’ll see - and you can bet your competitors will be doing the same.

Further reading

• Golden Image or Foil Ball? (Luke Kanies)
• Scripted Kickstart vs. Image-Based Deployment For Linux Systems (Michael DeHaan)

An earlier version of this article appeared on the Agile Sysadmin blog.

Pay attention at the back. Reductive Labs just announced Puppet training dates for London:

Puppet Training London: March 29 - April 2

There are two courses available: Becoming a Puppet Master (3 days) covers Puppet and Puppetmaster configuration, classes, modules, definitions, resource types, the resource abstraction layer, virtual resources, exported resources, stored configs, metaparameters, dependencies, events, tags, environments, Puppet language patterns and best practices. There’ll be a mix of teaching and interactive class exercises using your own laptop and virtual machines. It costs £1595 for the 3 day course.

If you’re a seasoned old Puppet hand and you know all that stuff by heart, then the Puppet Developer course (2 days) is probably for you. You’ll get an introduction to Ruby for Puppet, as well as learning how to craft your own custom resource types and providers, develop custom functions and Facter facts. In addition there’s a segment on testing practices and RSpec for Puppet. The cost is £995 for two days.

Register for the training (get £100 off each course if you register before March 15th) or contact Scott Campbell to find out more. And don’t forget to bring an apple for the teacher.

“Is the web site down?” asked my boss. Probably the five words I least want to hear, along with “I’ve run over your cat” and “I’ll pay you next month”.

Yes, the site was down - I’d made a minor Puppet change, and assumed this would roll out without any problems. Of course, the problem with assumptions is that they make an ASS out of U in front of your BOSS. Whenever I use Puppet to make changes to production servers, I like to do it by running Puppet manually, rather than waiting for the half-hourly auto-run, and I like to do a dry run first to see what’s going to happen.

Puppet’s dry-run feature is a powerful tool that’s often overlooked by busy sysadmins. Even if you test your Puppet manifests on a virtualised replica of your production site, which many people don’t have the time or the budget to do, pushing changes out live can have unforeseen side effects which are best avoided.

To dry-run Puppet, use the --noop flag:

# puppetd --test --noop
info: Caching catalog at /var/puppet/state/localconfig.yaml
notice: Starting catalog run
762a763
> This change will screw up your Apache server.
notice: //Node[soupnazi]/nagios::server/apache/File[/etc/httpd/conf/httpd.conf]/source: 
  is {md5}327dc6b2d2337e4467840c007a0edd61, should be puppet:///apache/httpd.conf (noop)
info: //Node[soupnazi]/nagios::server/apache/File[/etc/httpd/conf/httpd.conf]: Scheduling refresh 
  of Service[httpd]
notice: //Node[soupnazi]/nagios::server/apache/Service[httpd]: Would have triggered refresh from 1 
  dependencies
notice: Finished catalog run in 8.93 seconds

Puppet’s ‘noop’ (no-operation) mode shows you what would happen, but doesn’t actually do it. As you can see, Puppet reports that it would have updated the httpd.conf file and restarted Apache, and it helpfully shows us the line it would have inserted into the file:

> This change will screw up your Apache server.

Aren’t you glad you added the --noop flag? Of course, whenever you’re contemplating rolling out changes to a live server like this, you’ll want to stop the Puppet daemon running on the box temporarily (or disable the cron job if you run Puppet out of cron). The last thing you want is Puppet running automatically in “really do it” mode while you’re scrambling to fix the broken manifest.

Puppet’s dry run mode is not perfect - resource Y may fail because it requires resource X, for example, and resource X has not really been applied. Puppet isn’t so clever as to model the state of the box if each resource had been really applied - which is not surprising, because that’s a hard problem. Luckily, it doesn’t need to be perfect: dry-running Puppet in this way will catch a lot of small but potentially disastrous errors before they happen.

Opinions differ as to how meaningful or correct a dry-run mode can be, since changing the state of the box makes things unpredictable (Chef doesn’t have a dry run option at all), but I’ll take avoiding a disaster over academic arguments any day.

“Thanks for getting the site back up,” said my boss. “Now, something else I meant to tell you. I’ll pay you next month.” Read more »