Skip navigation.
Home

puppet

Puppet 5 Beginner's Guide (third edition)

Order your copy now on Amazon.com (Amazon.co.uk)

A new book on Puppet for the cloud-native era

A new edition of the bestselling Puppet Beginner’s Guide is now available, covering all the latest features of Puppet 5. Learn Puppet from scratch, from installation to managing complex infrastructures, including chapters on managing Amazon AWS cloud resources and Docker containers with Puppet. Every code example is available right now online in a special GitHub repo just for this book. There is also a complete, working Puppet infrastructure for you to download, adapt, and use on your own servers.

Praise for the Puppet Beginner’s Guide

Reviews of this edition:

“I’ve been using Puppet for some time now, and picked this book up after many great recommendations across the internet. Though this is titled as a Beginner’s Guide, I’ve found that my own Puppet knowledge has increased, and I’ve started rewriting and making use of a lot of the concepts introduced in this book, some of which are idiomatic. I would recommend this book to anyone looking for a guide on how to use Puppet from start to finish, whether you ‘think you know it all’, or whether you are just curious.” - Amazon.co.uk

“Excellent Puppet primer for any SRE. As a managed service provider, we recently migrated a lot of workloads to being managed by Puppet. This includes Amazon instances across multiple regions for several customers. The examples for AWS management provided in the book definitely improved on the structure of the manifests we built from just assembling bits and pieces from Stack Overflow. Another very positive surprise was to discover how Puppet is also able to orchestrate our Docker workloads throughout their lifecycle. We previously relied on a pipeline on GitLab CI, but being able to consolidate those into Puppet streamlines our work further. This book is a phenomenal encyclopedia for looking up the appropriate syntax and tons of best practices for everything you need to know about Puppet. The structure of the book makes digesting such a complex topic extremely tolerable, and I can only give it my highest recommendation for anyone aspiring to learn or improve their Puppet skills.” - Amazon.com

Reviews of previous editions:

“This book does not just cover Puppet well, but I was pleasantly surprised to find that it also goes into several related topics. e.g. using Git to keep track of changes. There is also a chapter on containers (and Docker), some issues with them and how to use Puppet to manage them. After that there’s a chapter on cloud resources, in particular AWS. Again the book provides a lot of useful information on how to manage various resources like EC2 instances, VPCs, etc.” - Amazon.com

“I’ve thoroughly enjoyed it. It explains the subject matter very well and takes a lot of the mystery out of Puppet. If you are looking for a way into Puppet I highly recommend this book.” - Amazon.co.uk

“Great guide that’s easy to read and written well so the concepts are simple to understand.” - Amazon.co.uk

“Very clear explanations.” - Amazon.com

“Well-written… smoothly-structured… John Arundel’s new book is a solid guide to helping you become both a proficient Puppet user and a more efficient, knowledgeable, and versatile system administrator.” — Slashdot

“I can’t imagine having found anything better. This book avoids the stress and boredom of a slow-paced learning session by diving right in to the most important concepts. It was all I needed to feel confident at work, and it only took an hour before I was writing my own code.” — Amazon.com

“John Arundel is both a superb engineer and just as importantly an excellent teacher/communicator. Currently the best Puppet book available and essential for anyone wanting to learn Puppet.” — Amazon.com (Kindle edition)

A word from the author

There’s only so much you can learn from books. As Proust wrote, “We don’t receive wisdom; we must discover it for ourselves after a journey that no one can take for us or spare us.”

It’s good to have a friend point us in the right direction and come with us a little way for moral support, but then we need to walk on by ourselves. I hope that this book will be the beginning of your journey, not the end.

The world-famous classical guitarist John Williams was once asked how long it took him to learn to play the guitar. “I’m still learning,” he said.

Audience

This book is designed for those new to Puppet, including system administrators and developers who are looking to manage computer server systems for configuration management. No prior programming or system administration experience is assumed.

Mission

The Puppet 5 Beginner’s Guide gets you up and running with Puppet 5 straight away with real-world examples. It will take you from complete beginner to confident Puppet user with a series of clear examples and case studies. By reading this book, you will learn to manage every aspect of your server setup. You will gain expert understanding of Puppet’s latest and most advanced features and put them to work.

Objectives and achievements

This book helps you to:

  • Install and set up Puppet and discover the latest and most advanced features
  • Automate your IT infrastructure
  • Use the latest features in Puppet 5+ and official modules
  • Manage clouds, containers, and orchestration
  • Learn best practices to make Puppet more reliable and high performing

General structure

  1. Getting started with Puppet
  2. Creating your first manifests
  3. Managing Puppet code with Git
  4. Understanding Puppet resources
  5. Variables, expressions, and facts
  6. Managing data with Hiera
  7. Mastering modules
  8. Classes, roles, and profiles
  9. Managing files with templates
  10. Controlling containers
  11. Orchestrating cloud instances
  12. Putting it all together

Detailed outline

About this book

A brief introduction to the aims and methods of the book, and how to use it.

1. Getting started with Puppet

In this chapter you’ll learn about some of the challenges of managing configuration on servers, some common solutions to this problem, and how automation tools such as Puppet can help. You’ll also learn how to download the GitHub repository containing all the source code and examples in this book, how to set up your own Vagrant virtual machine to run the code, and how to download and install Puppet.

  • Why do we need Puppet anyway?
    • Keeping configuration synchronized
    • Repeating changes across many servers
    • Self-updating documentation
    • Why not just write shell scripts?
    • Why not just use containers?
    • Why not just use serverless?
    • Configuration management tools
  • What is Puppet?
    • Resources and attributes
    • Puppet architectures
  • Getting ready for Puppet
    • Getting the Git repo
    • Installing Virtualbox and Vagrant
    • Running your Vagrant VM
    • Alternative Vagrant VMs
    • Troubleshooting Vagrant

2. Creating your first manifests

The Puppet manifest is the complete expression, in code form, of how your server should be configured. It consists of a set of resources, each of which models some aspect of system configuration, such as a file, or a software package.

In this chapter you’ll learn how to write your first manifest with Puppet, and how to put Puppet to work configuring a server. You’ll also understand how Puppet reads and applies a manifest. You’ll see how to use Puppet to manage the contents of files, how to install packages, and how to control services.

  • Hello, Puppet: your first Puppet manifest
    • Understanding the code
    • Modifying existing files
    • Dry-running Puppet
    • How Puppet applies the manifest
    • Creating a file of your own
  • Managing packages
    • How Puppet applies the manifest
    • Exercise
    • Querying resources with puppet resource
  • Services
    • Getting help on resources with puppet describe
    • The package-file-service pattern
    • Understanding the code
    • Notifying a linked resource
    • Resource ordering with require

3. Managing Puppet code with Git

In this chapter we’ll install Git, create a repo, and import the manifests created in previous chapters. Readers will learn to make some changes to the manifests and commit them to the repo. Readers will learn how to check the revision history of the repo. We will clone our existing repo to a master repo, and deploy it on a new server. We will set up a script and cron job to pull changes from the master repo and apply them automatically.

  • What is version control?
    • Tracking changes
    • Sharing code
  • Creating a Git repo
    • Making your first commit
    • How often should I commit?
    • Branching
  • Distributing Puppet manifests
    • Creating a GitHub account and project
    • Pushing your repo to GitHub
    • Cloning the repo
  • Fetching and applying changes automatically
    • Writing a manifest to set up regular Puppet runs
    • Applying the run-puppet manifest
    • The run-puppet script
    • Testing automatic Puppet runs
    • Managing multiple nodes

4. Understanding Puppet resources

We’ve already met three important types of Puppet resource: packages, files, and services. In this chapter we’ll go into a little more detail on these to see how to exploit their power to the full. We’ll also learn about three other important resource types: user, cron, and exec resources. Along the way we’ll talk about:

  • Files
    • The path attribute
    • Managing whole files
    • Ownership
    • Permissions
    • Directories
    • Trees of files
    • Symbolic links
  • Packages
    • Uninstalling packages
    • Installing specific versions
    • Installing the latest version
    • Installing Ruby gems
    • Installing gems in Puppet’s context
    • Using ensure_packages
  • Services
    • The hasstatus attribute
    • The pattern attribute
    • The hasrestart and restart attributes
  • Users
    • Creating users
    • The user resource
    • The group resource
    • Managing SSH keys
    • Removing users
  • Cron resources
    • Attributes of the cron resource
    • Randomizing cron jobs
    • Removing cron jobs
  • Exec resources
    • Automating manual interaction
    • Attributes of the exec resource
    • The user attribute
    • The onlyif and unless attributes
    • The refreshonly attribute
    • The logoutput attribute
    • The timeout attribute
    • How not to misuse exec resources

5. Variables, expressions, and facts

In this chapter you will learn about variables in Puppet, and discover the different types of data structures and data types that Puppet provides. You will find out how to use variables and values in expressions, and how to apply different resources depending on the value of expressions, using conditional statements. You will also learn how Puppet manifests can get data about the server using Facter, find out which are the most important standard facts, and see how to create your own custom facts. Finally, you will see how to use the each function to loop over arrays and hashes.

  • Introducing variables
    • Using Booleans
    • Interpolating variables in strings
    • Creating arrays
    • Declaring arrays of resources
    • Understanding hashes
    • Setting resource attributes from a hash
  • Introducing expressions
    • Meeting Puppet’s comparison operators
    • Introducing regular expressions
    • Using conditional expressions
    • Making decisions with if statements
    • Choosing options with case statements
  • Finding out facts
    • Using the facts hash
    • Running the facter command
    • Accessing hashes of facts
    • Referencing facts in expressions
    • Using memory facts
    • Discovering networking facts
    • Providing external facts
    • Creating executable facts
  • Iterating over arrays
    • Using the each function
    • Iterating over hashes

6. Managing data with Hiera

In this chapter you will learn why it’s useful to separate program logic from configuration data. You will see how to use Puppet’s built-in Hiera mechanism to store and query config data, including encrypted secrets such as passwords and SSL keys. You will also learn how to use Hiera data to construct Puppet resources directly.

  • Why Hiera?
    • Data needs to be maintained
    • Settings depend on nodes
    • Operating systems differ
    • The Hiera way
  • Setting up Hiera
  • Adding Hiera data to your Puppet repo
    • Troubleshooting Hiera
  • Querying Hiera
    • Typed lookups
  • Types of Hiera data
    • Single values
    • Boolean values
    • Arrays
    • Hashes
  • Interpolation in Hiera data
    • Using lookup()
    • Using alias()
    • Using literal()
  • The hierarchy
    • Dealing with multiple values
    • Merge behaviours
    • Data sources based on facts
    • What belongs in Hiera?
  • Creating resources with Hiera data
    • Building resources from Hiera arrays
    • Building resources from Hiera hashes
    • The advantages of managing resources with Hiera data
  • Managing secret data
    • Setting up GnuPG
    • Adding an encrypted Hiera source
    • Creating an encrypted secret
    • How Hiera decrypts secrets
    • Editing or adding encrypted secrets
    • Distributing the decryption key

7. Mastering modules

In this chapter you will learn about Puppet Forge, the public repository for Puppet modules, and you’ll see how to install and use third-party modules from Puppet Forge with the standard module management tool r10k. You’ll work through examples of using three key Forge modules: puppetlabs-apache, puppetlabs-mysql, and puppet-archive. Finally, you’ll learn how to develop your own Puppet module from scratch, working through a complete example, how to add appropriate metadata for your module, and how to upload it to Puppet Forge.

  • Using Puppet Forge modules
    • What is the Puppet Forge?
    • Finding the module you need
    • Using r10k
    • Understanding the Puppetfile
    • Managing dependencies with generate-puppetfile
  • Using modules in your manifests
    • Using puppetlabs/mysql
    • Using puppetlabs/apache
    • Using puppet/archive
  • Exploring the standard library
    • Safely installing packages with ensure_packages
    • Modifying files in place with file_line
    • Introducing some other useful functions
    • The pry debugger
  • Writing your own modules
    • Creating a repo for your module
    • Writing the module code
    • Creating and validating the module metadata
    • Tagging your module
    • Installing your module
    • Applying your module
    • More complex modules
    • Uploading modules to the Puppet Forge

8. Classes, roles, and profiles

In this chapter you will explore the details of Puppet classes, the distinction between defining a class and creating instances of that class, and how to pass parameters to classes. You’ll learn how to create defined resource types, and how they differ from classes. You’ll also see how to organize your Puppet code using the concepts of roles, and profiles, and how to use Hiera data to assign different classes to nodes.

  • Classes
    • The class keyword
    • Declaring parameters to classes
    • Automatic parameter lookup from Hiera data
  • Parameter data types
    • Available data types
    • Content type parameters
    • Range parameters
    • Flexible data types
  • Defined resource types
  • Type aliases
  • Managing classes with Hiera
    • Using include with lookup()
    • Common and per-node classes
  • Roles and profiles
    • Roles
    • Profiles

9. Managing files with templates

In this chapter we’ll learn about an important and powerful feature of Puppet: the template. We’ll see how to use a simple template to interpolate the values of Puppet variables, facts, and Hiera data into a file, and we’ll also introduce more complex templates using iteration and conditional statements to generate dynamic configuration files.

  • What are templates?
    • The dynamic data problem
    • Puppet template syntax
  • Using templates in your manifests
    • Referencing template files
    • Inline templates
    • Template tags
    • Computations in templates
    • Conditional statements in templates
  • Iteration in templates
    • Iterating over Facter data
    • Iterating over structured facts
    • Iterating over Hiera data
  • Working with templates
    • Passing parameters to templates
    • Validating template syntax
    • Rendering templates on the command line
  • Legacy ERB templates

10. Controlling containers

In this chapter we’ll look at the emerging topic of containers and see how it relates to configuration management. We’ll see how to use Puppet to manage Docker itself, as well as images and containers, and explore some different strategies for managing configuration within containers.

  • Understanding containers
    • The deployment problem
    • Options for deployment
    • Introducing the container
    • What Docker does for containers
    • Deployment with Docker
    • Building Docker containers
    • The layered filesystem
    • Managing containers with Puppet
  • Managing Docker with Puppet
    • Installing Docker
    • Running a Docker container
    • Stopping a container
    • Running multiple instances of a container
  • Managing Docker images
    • Building images from Dockerfiles
    • Managing Dockerfiles
  • Building dynamic containers
    • Configuring containers with templates
    • Self-configuring containers
  • Persistent storage for containers
    • Host-mounted volumes
    • Docker volumes
  • Networking and orchestration
    • Connecting containers
    • Container orchestration
    • What is orchestration?
    • What orchestration tools are available?
  • Running Puppet inside containers
    • Are containers mini VMs or single processes?
    • Configuring containers with Puppet
    • Containers need Puppet too

11. Orchestrating cloud resources

In this chapter we will learn how to use Puppet to create and manage AWS cloud resources, such as EC2 instances.

  • Introducing the cloud
    • Automating cloud provisioning
    • Using CloudFormation
    • Using Terraform
    • Using Puppet
  • Setting up an Amazon AWS account
    • Creating an AWS account
    • Creating an IAM policy
    • Creating an IAM user
    • Storing your AWS credentials
  • Getting ready to use puppetlabs/aws
    • Creating a key pair
    • Installing the puppetlabs/aws module
    • Installing the AWS SDK gem
  • Creating EC2 instances with Puppet
    • Choosing an Amazon Machine Image (AMI)
    • Creating the EC2 instance
    • Accessing your EC2 instance
  • VPCs, subnets, and security groups
    • The ec2_securitygroup resource
    • The ec2_instance resource
  • Managing custom VPCs and subnets
    • Creating an instance in a custom VPC
    • The ec2_vpc resource
    • The ec2_vpc_internet_gateway resource
    • The ec2_vpc_routetable resource
    • The ec2_vpc_subnet resource
    • Other AWS resource types
  • Provisioning AWS resources from Hiera data
    • Iterating over Hiera data to create resources
    • Orchestrator nodes
    • Cleaning up unused resources

12. Putting it all together

In this chapter we will apply all the lessons from previous chapters to see what a complete, working infrastructure looks like. This is a complete worked example which readers can use as the basis of their own Puppet codebase, adapting and expanding it as needed.

  • Getting the demo repo
    • Copying the repo
  • Understanding the demo repo
    • The control repo
    • Module management
    • Classes
    • Roles
    • Profiles
    • Users and access control
    • SSH configuration
    • Sudoers configuration
    • Time zone and clock synchronization
    • Puppet configuration
  • The bootstrap process
  • Adapting the repo for your own use
    • Configuring users
    • Adding per-node data files and role classes
    • Modifying the bootstrap credentials
  • Bootstrapping a new node
    • Bootstrapping a Vagrant VM
    • Bootstrapping physical or cloud nodes
    • Using other distributions and providers
  • The beginning

Author Bio

John Arundel is a devops consultant, which means he helps people build world-class web operations teams and infrastructure, and has fun doing it. He was formerly a senior operations engineer at global telco Verizon, designing resilient, high-performance infrastructures for major corporations like Ford, McDonald’s, and Bank of America. He is now an independent consultant, working closely with selected clients to deliver web-scale performance and enterprise-grade resilience on a startup budget.

He likes writing books, especially about Puppet (The Puppet Cookbook is available from the same publisher). It seems that at least some people enjoy reading them, or maybe they just like the pictures. He also provides training and coaching on Puppet and devops, which it turns out is far harder than simply doing the work himself.

Off the clock, he is a medal-winning competitive rifle and pistol shooter, and a decidedly uncompetitive piano player. He lives in a small cottage in Cornwall, England and believes, like Cicero, that if you have a garden and a library, then you have everything you need.

You may like to follow him on Twitter at @bitfield.

Acknowledgments

My grateful thanks are due to Jo Rhett, who made innumerable improvements and suggestions to this book, and whose Puppet expertise and clarity of writing I can only strive to emulate. Also to the original Puppet master, Luke Kanies, who created a configuration management tool that sucks less, and my many other friends at Puppet. Many of the key ideas in this book came from them and others including Przemyslaw ‘SoboL’ Sobieski, Peter Bleeck, and Igor Galić.

The techniques and examples in the book come largely from real production codebases, of my consulting clients and others, and were developed with the indispensable assistance of my friends and colleagues Jon Larkowski, Justin Domingus, Walter Smith, Ian Shaw, and Mike Thomas. Special thanks are also due to the Perseids Project at Tufts University, and most of all to the inestimable Bridget Almas, who patiently read and tested everything in the book several times and made many valuable suggestions, not to mention providing continuous moral support, love, and guidance throughout the writing process. This book is for her.

Puppet and MySQL: create databases and users

Submitted by John Arundel on Mon, 09/13/2010 - 15:46

MySQL module for Puppet

Puppet MySQL management couldn’t be easier. Most applications use some kind of SQL database, and MySQL is a simple, easy to use, drop-in solution. In this article I’ll show you how to manage your MySQL servers, users, databases, and access permissions using Puppet. Read more »

»

Puppet Tutorial for Linux part 2: Client and Server

Submitted by John on Mon, 04/26/2010 - 15:39

Puppet tutorial series

In Part 1 of this Puppet tutorial we saw how to install Puppet on a Linux machine from source, and create a basic manifest which controls the NTP service. In this episode we’ll cover setting up a Puppet server and then using it to control multiple client machines.

Installing servers with Puppet

In the first part of the tutorial, we installed Puppet directly from the source. This is a great way to learn and experiment, but for production purposes we would like to use a standard package - in this case, an Ubuntu package we can install via apt-get. This way, you can ensure that the same version of Puppet is present on your servers and update it automatically if need be. The Ubuntu package also has a few bonus features that the source package doesn’t.

If you want to follow along with the tutorial using your existing Puppet source install, that’s fine. It’s a good idea to make sure you’re using the same version of Puppet on both client and server, or you may run into obscure and annoying problems, so either install from the same source package on your client machine, or set up two fresh machines with the Ubuntu package as described below.

The version of Puppet in the standard Ubuntu repositories is a little old, so add Mathias Gug’s puppet-backports repository to your machine so that you can get a more recent version:

# add-apt-repository ppa:mathiaz/puppet-backports
# apt-get update
# apt-get install puppet puppetmaster

You can now start the Puppetmaster daemon and make the service persistent:

# update-rc.d puppetmaster defaults
# puppet master --mkusers

If there is a firewall between the Puppetmaster and its clients (eg an iptables firewall on the Puppetmaster server itself) you will need to open TCP port 8140 to the clients.

Create a client manifest

In Part 1 of the Puppet Tutorial we created a file /etc/puppet/manifests/nodes.pp, which lists the nodes (machines) that Puppet knows about, and what configuration they should have. Currently it looks like this:


node myserver {
    include ntp
}

As we’re adding a new node to the system, we need to modify the file so that it reads like this:


node myserver {
    include ntp
}

node myclient {
    include ntp
}

Authorising a client

Puppet uses SSL (Secure Sockets Layer), an encrypted protocol, to communicate between master and clients. This means that only a client with a correctly signed SSL certificate can access the Puppetmaster and receive its configuration. To exchange certificates between the master and client, follow this procedure.

Configuring the client to contact the server

Edit your /etc/puppet/puppet.conf file to tell the client where to find the Puppetmaster:

server = myserver.mydomain.com

Generate a certificate request

On the client, run:

# puppet agent --test
info: Creating a new certificate request for myclient.mydomain.com
info: Creating a new SSL key at /etc/puppet/ssl/private_keys/myclient.mydomain.com.pem
warning: peer certificate won't be verified in this SSL session
notice: Did not receive certificate
notice: Set to run 'one time'; exiting with no certificate

Sign the certificate

On the master:

# puppetca -l
myclient.mydomain.com

There is a certificate request pending for myclient. To sign it, run:

# puppetca -s myclient.mydomain.com
notice: Signed certificate request for myclient.mydomain.com
notice: Removing file Puppet::SSL::CertificateRequest myclient.mydomain.com at 
  '/var/lib/puppet/ssl/ca/requests/myclient.mydomain.com.pem'

If there is no certificate request listed, the client was not able to contact the server for some reason. Check that you have the right server address set in puppet.conf on the client, and that TCP port 8140 is open on both the master and client firewalls. The puppet master daemon also needs to be running on the master.

Run Puppet for the first time

On the client, you should now be able to run:

# puppet agent --test
notice: Got signed certificate
info: Caching catalog at /var/puppet/state/localconfig.yaml
notice: Starting catalog run
notice: //ntp/Service[ntp]/ensure: ensure changed 'stopped' to 'running'
notice: Finished catalog run in 0.92 seconds

Just as on the server in the first part of the tutorial, Puppet is now applying the ntp class to myclient. At the moment all this class does is ensure the ntp service is running, but of course it could do many more interesting things. And that’s what we’ll look at next time in Part 3.

Puppet books

If you’re excited about Puppet and want to learn more, may I modestly recommend the Puppet 2.7 Cookbook? The Puppet 2.7 Cookbook takes you beyond the basics to explore the full power of Puppet, showing you in detail how to tackle a variety of real-world problems and applications. At every step it shows you exactly what commands you need to type, and includes full code samples for every recipe.

»

What's wrong with this image?

Submitted by John on Thu, 04/01/2010 - 18:11

The problem

Is this a familiar nightmare? “The server you built yesterday is great,” your boss enthuses. “Now we need a hundred more just like it. And when you’ve done that, could you take a look at my laptop before lunch? I can’t seem to get into my email.”

It’s common nowadays for sysadmins to manage very large numbers of servers - often virtual servers created on-demand in cloud platforms like Amazon EC2. How do you build so many servers and keep them updated - automatically, reliably, and quickly?

One common strategy is the golden image pattern: set up the server the way you want it, and then copy the disk image and use that to boot as many clone machines as you need. Another is the automation pattern: start with a generic server, and then apply automatic configuration tools such as Puppet to bring the machine into a specified state where it is ready for production.

The golden image

The golden image is attractive to many because it’s easy and cheap. It’s quick to create images, and you just need a big disk to store them.

Imaged machines also have low latency: they’re ready for production in just a few minutes from startup. Finally, it’s platform agnostic: you can image virtually any kind of machine, even Windows (which most config automation tools can’t handle very well).


So what’s the problem with the golden image? Luke Kanies, creator of Puppet, prefers to call it the “foil ball” approach: your images can quickly become an unmanageable, crumpled mess. Firstly, how many images do you need? Well, as many as you have different types of machines: web servers, databases, load balancers, DNS servers, and so on.

The number of images multiplies again when you take history into account. What happens when you need to recreate the state of the database server last month to troubleshoot a problem? You need to keep old images around, which means a new image every time you make a change.

And here’s a thing: if you do need to make a change (for example a security upgrade for a critical package), you’ll have to rebuild the golden image and then re-image and reboot all affected machines - which means lots of bandwidth and possibly downtime. And, unfortunately, even imaged machines usually need some final tweaking (eg IP addresses) before they’re ready for use.

Automation tools

By comparison, building your machines with an automatic configuration tool such as Puppet, Chef or even cfengine gives you many advantages. The configuration is self-documenting; it’s easy to see what should be on each machine, and more importantly, why it’s there. It will also stay in sync with reality.

Although your server types will be different, they will also likely have a lot in common. With images, these supposedly common settings could easily drift apart over time. With automation tools, you can put these settings into modules or classes, just as a software developer re-uses a module or library in another project.

In fact, all the advantages of software modules are available: you can share code with others, download public recipes and manifests, and maintain a complete version history of your network.

Automated configuration is intrinsically multi-platform. Puppet and Chef can abstract away some of the differences between operating systems, and you can add minor tweaks of your own. For example, you might need a slightly different MySQL config setting for MySQL 4.x as opposed to 5.x. A one-line switch in the configuration can address this problem, instead of creating two disk images.

Of course, everything comes with a downside. It is more difficult and usually more time-consuming to build an automated configuration, especially from scratch and when you’re still learning to use the tools. And some things are hard to automate. Some software just wasn’t written with automated installation and management in mind.

Also, not all software will be packaged for your platform (or you need special compilation options). In these cases you will need to package the software yourself.

Finally, configuration management doesn’t always mix well with human intervention. In particular, someone may make a config change on the machine which is then automatically reversed by Puppet. At best this can be a nasty surprise, at worst it can result in a site outage. Managing desktops automatically can create similar problems. This needs to be addressed at a business level with a proper change control process and making it clear to everyone where the boundaries of automatic management lie.

Combined strategies

So both imaging and automation have their place; neither is a magic bullet. The best strategy for scaling your deployments is often a judicious combination of these two approaches. You can, for example, start with a ‘stem cell’ image containing an approved base build, security fixes and settings which don’t often change, which is then configured by Puppet or Chef. Or you could use provisioning tools such as Kickstart and Cobbler and then hand over to Puppet. Or even build the machine with Puppet and then create a golden image from this.

One thing is certain: ignoring the problem is not an option. Moving to an automated network might seem like a big project, but the key is to start small. Set aside a block of time every week to spend on improving your infrastructure, and automate one small thing at a time as part of a gradual continuous process. The more you invest in this, the more benefits you’ll see - and you can bet your competitors will be doing the same.

Further reading

An earlier version of this article appeared on the Agile Sysadmin blog.

»

Can you master Puppet?

Submitted by John Arundel on Sat, 03/06/2010 - 14:15

Pay attention at the back. Reductive Labs just announced Puppet training dates for London:

Puppet Training London: March 29 - April 2

There are two courses available: Becoming a Puppet Master (3 days) covers Puppet and Puppetmaster configuration, classes, modules, definitions, resource types, the resource abstraction layer, virtual resources, exported resources, stored configs, metaparameters, dependencies, events, tags, environments, Puppet language patterns and best practices. There’ll be a mix of teaching and interactive class exercises using your own laptop and virtual machines. It costs £1595 for the 3 day course.

If you’re a seasoned old Puppet hand and you know all that stuff by heart, then the Puppet Developer course (2 days) is probably for you. You’ll get an introduction to Ruby for Puppet, as well as learning how to craft your own custom resource types and providers, develop custom functions and Facter facts. In addition there’s a segment on testing practices and RSpec for Puppet. The cost is £995 for two days.

Register for the training (get £100 off each course if you register before March 15th) or contact Scott Campbell to find out more. And don’t forget to bring an apple for the teacher.

»

Puppet dry run

Submitted by John Arundel on Tue, 03/02/2010 - 19:16

“Is the web site down?” asked my boss. Probably the five words I least want to hear, along with “I’ve run over your cat” and “I’ll pay you next month”.

Yes, the site was down - I’d made a minor Puppet change, and assumed this would roll out without any problems. Of course, the problem with assumptions is that they make an ASS out of U in front of your BOSS. Whenever I use Puppet to make changes to production servers, I like to do it by running Puppet manually, rather than waiting for the half-hourly auto-run, and I like to do a dry run first to see what’s going to happen.

Puppet’s dry-run feature is a powerful tool that’s often overlooked by busy sysadmins. Even if you test your Puppet manifests on a virtualised replica of your production site, which many people don’t have the time or the budget to do, pushing changes out live can have unforeseen side effects which are best avoided.

To dry-run Puppet, use the --noop flag:

# puppetd --test --noop
info: Caching catalog at /var/puppet/state/localconfig.yaml
notice: Starting catalog run
762a763
> This change will screw up your Apache server.
notice: //Node[soupnazi]/nagios::server/apache/File[/etc/httpd/conf/httpd.conf]/source: 
  is {md5}327dc6b2d2337e4467840c007a0edd61, should be puppet:///apache/httpd.conf (noop)
info: //Node[soupnazi]/nagios::server/apache/File[/etc/httpd/conf/httpd.conf]: Scheduling refresh 
  of Service[httpd]
notice: //Node[soupnazi]/nagios::server/apache/Service[httpd]: Would have triggered refresh from 1 
  dependencies
notice: Finished catalog run in 8.93 seconds

Puppet’s ‘noop’ (no-operation) mode shows you what would happen, but doesn’t actually do it. As you can see, Puppet reports that it would have updated the httpd.conf file and restarted Apache, and it helpfully shows us the line it would have inserted into the file:

> This change will screw up your Apache server.

Aren’t you glad you added the --noop flag? Of course, whenever you’re contemplating rolling out changes to a live server like this, you’ll want to stop the Puppet daemon running on the box temporarily (or disable the cron job if you run Puppet out of cron). The last thing you want is Puppet running automatically in “really do it” mode while you’re scrambling to fix the broken manifest.

Puppet’s dry run mode is not perfect - resource Y may fail because it requires resource X, for example, and resource X has not really been applied. Puppet isn’t so clever as to model the state of the box if each resource had been really applied - which is not surprising, because that’s a hard problem. Luckily, it doesn’t need to be perfect: dry-running Puppet in this way will catch a lot of small but potentially disastrous errors before they happen.

Opinions differ as to how meaningful or correct a dry-run mode can be, since changing the state of the box makes things unpredictable (Chef doesn’t have a dry run option at all), but I’ll take avoiding a disaster over academic arguments any day.

“Thanks for getting the site back up,” said my boss. “Now, something else I meant to tell you. I’ll pay you next month.” Read more »

»

Puppet resources

Puppet is an indispensable configuration management and Linux automation tool for the system administrator who prefers to spend her time improving the infrastructure, rather than doing tedious manual labour. Chef, Puppet, Fabric and other tools can be a huge boost to productivity. From small beginnings, you can grow your Puppet installation to manage a whole network, or many networks. However, with great power comes a steep learning curve. There are lots of great resources and tutorials available if you want to learn Puppet - here’s a brief guide for aspiring devops.

Puppet help

Need an introduction to Puppet? Start here.

What is Puppet?

Read more »
»

Puppet Drupal recipes

Submitted by John on Fri, 02/05/2010 - 16:24

Drupal, Puppet. Puppet, meet Drupal

Puppet and Drupal make a great combination. Drupal is an amazing tool for quickly constructing attractive, functional web sites. It lets you manage large numbers of web sites from a single installation, and (via add-on modules) provides almost any CMS or blog feature you could want.

However, like any powerful tool, Drupal takes some learning. It also needs a certain amount of discipline to manage Drupal servers without getting into a chaotic mess. The Drupal sysadmin can end up trying to navigate a spaghetti of ad-hoc symlinks and face problems upgrading, maintaining, monitoring and backing up a large Drupal installation. Aegir can help with this (I’ll look at Aegir vs. Puppet in a future article) but first we need to get Drupal itself under control.

Fortunately, Puppet can help you tame Drupal and use the power of configuration management to bring your Drupal sites under control. In this article I’ll explain some techniques and Puppet recipes I use to manage Drupal sites and servers, including my own sites, including this one! Read more »

»

Scaling Puppet with Git

Submitted by Stephen Nelson-Smith on Wed, 01/27/2010 - 02:29

Scaling Puppet with Git

More and more people are turning to systems automation tools like Puppet and Chef to get the most out of their environments, and to create time to focus on delivering business benefits. Scaling Puppet is most commonly done using client/server mode, in which every client is issued with an SSL certificate, and conversations take place between clients and a server, over HTTP, and manifests and assets are served over the network, and applied by a locally running Puppet daemon. However, is there a better way? We present an alternative to the traditional Puppetmaster solution which we like to call ‘Git Puppet’.

Guest article by Stephen Nelson-Smith

Update: There is a fuller and more up-to-date description of my Git-based Puppet infrastructure in The Puppet Cookbook - so after you read this introduction, you may find it helpful to read the book too. Read more »

»

Puppet versus Chef: 10 reasons why Puppet wins

Submitted by John on Tue, 01/12/2010 - 20:32

Puppet, Chef, cfengine, and Bcfg2 are all players in the configuration management space. If you’re looking for Linux automation solutions, or server configuration management tools, the two technologies you’re most likely to come across are Puppet and Opscode Chef. They are broadly similar in architecture and solve the same kinds of problems. Puppet, from Reductive Labs, has been around longer, and has a large user base. Chef, from Opscode, has learned some of the lessons from Puppet’s development, and has a high-profile client: EngineYard.

You have an important choice to make: which system should you invest in? When you build an automated infrastructure, you will likely be working with it for some years. Once your infrastructure is already built, it’s expensive to change technologies: Puppet and Chef deployments are often large-scale, sometimes covering thousands of servers.

Chef vs. Puppet is an ongoing debate, but here are 10 advantages I believe Puppet has over Chef today. Read more »

»
Syndicate content