Skip navigation.

About Bitfield Consulting

John Arundel, principal consultant, author, and devops expert, has been writing software for 35 years, managing Unix systems for two decades, and hacking on infrastructure from nuclear power stations to Netflix since he was knee-high to a login prompt. He has been using Puppet in commercial web operations for over 10 years and has deployed Puppet solutions for dozens of companies from Singapore to Seattle, and at fairly large scales (many thousands of servers in a single site). He is also the author of The Puppet Cookbook and The Puppet Beginner’s Guide, which have sold more than 15,000 copies, and regularly provides coaching and training on Puppet from beginner to enterprise level.

John also regularly regales his 100,000+ Twitter followers with deep thoughts about computing, configuration management, and software development at @bitfield. Even though no-one can agree what ‘devops’ means, it seems to be a partial description of what he does.

He works closely with selected clients worldwide, consulting, coaching, and training on:

  • Building devops and operations teams
  • Building a friendly, collegial, supportive, diverse, and effective team culture
  • Building scalable, reliable, and high-performance infrastructure
  • Automating systems with Puppet
  • Development workflow with Git
  • Deploying apps and services with Docker
  • Virtualization, cloud infrastructure, and containerization

About John

Words from a few of my clients:

I’ve called upon John several times across my career to provide his wisdom and guidance on all things systems. He is an excellent Puppet consultant, and a diligent proponent of all things SysOps/DevOps. He has provided mentoring and support to our entire team, and is a fine ally and negotiator.”
—Mike Thomas

John is a pleasure to work with and helps us find order in the chaos. He offers expert, prompt and reliable service, and is flexible and willing to adjust to changing requirements and priorities. I recommend him without reservation.”
—Bridget Almas

Superb from both a technical and business angle, John makes everyone in the meeting room happy with his work.”
—James Darling

John combines strong technical expertise in both system administration and programming with excellent personal skills.”
—Stephen Nelson-Smith

His skills have saved my life more than once.”
—Dan Webb

Let’s talk

Could Bitfield Consulting help you? Email John at hello@bitfieldconsulting.com or message @bitfield on Twitter.

Puppet 5 Beginner's Guide (third edition)

Order your copy now on Amazon.com (Amazon.co.uk)

A new book on Puppet for the cloud-native era

A new edition of the bestselling Puppet Beginner’s Guide is now available, covering all the latest features of Puppet 5. Learn Puppet from scratch, from installation to managing complex infrastructures, including chapters on managing Amazon AWS cloud resources and Docker containers with Puppet. Every code example is available right now online in a special GitHub repo just for this book. There is also a complete, working Puppet infrastructure for you to download, adapt, and use on your own servers.

Praise for the Puppet Beginner’s Guide

Reviews of this edition:

I’ve been using Puppet for some time now, and picked this book up after many great recommendations across the internet. Though this is titled as a Beginner’s Guide, I’ve found that my own Puppet knowledge has increased, and I’ve started rewriting and making use of a lot of the concepts introduced in this book, some of which are idiomatic. I would recommend this book to anyone looking for a guide on how to use Puppet from start to finish, whether you ‘think you know it all’, or whether you are just curious.” - Amazon.co.uk

Excellent Puppet primer for any SRE. As a managed service provider, we recently migrated a lot of workloads to being managed by Puppet. This includes Amazon instances across multiple regions for several customers. The examples for AWS management provided in the book definitely improved on the structure of the manifests we built from just assembling bits and pieces from Stack Overflow. Another very positive surprise was to discover how Puppet is also able to orchestrate our Docker workloads throughout their lifecycle. We previously relied on a pipeline on GitLab CI, but being able to consolidate those into Puppet streamlines our work further. This book is a phenomenal encyclopedia for looking up the appropriate syntax and tons of best practices for everything you need to know about Puppet. The structure of the book makes digesting such a complex topic extremely tolerable, and I can only give it my highest recommendation for anyone aspiring to learn or improve their Puppet skills.” - Amazon.com

Reviews of previous editions:

This book does not just cover Puppet well, but I was pleasantly surprised to find that it also goes into several related topics. e.g. using Git to keep track of changes. There is also a chapter on containers (and Docker), some issues with them and how to use Puppet to manage them. After that there’s a chapter on cloud resources, in particular AWS. Again the book provides a lot of useful information on how to manage various resources like EC2 instances, VPCs, etc.” - Amazon.com

I’ve thoroughly enjoyed it. It explains the subject matter very well and takes a lot of the mystery out of Puppet. If you are looking for a way into Puppet I highly recommend this book.” - Amazon.co.uk

Great guide that’s easy to read and written well so the concepts are simple to understand.” - Amazon.co.uk

Very clear explanations.” - Amazon.com

Well-written… smoothly-structured… John Arundel’s new book is a solid guide to helping you become both a proficient Puppet user and a more efficient, knowledgeable, and versatile system administrator.” — Slashdot

I can’t imagine having found anything better. This book avoids the stress and boredom of a slow-paced learning session by diving right in to the most important concepts. It was all I needed to feel confident at work, and it only took an hour before I was writing my own code.” — Amazon.com

John Arundel is both a superb engineer and just as importantly an excellent teacher/communicator. Currently the best Puppet book available and essential for anyone wanting to learn Puppet.” — Amazon.com (Kindle edition)

A word from the author

There’s only so much you can learn from books. As Proust wrote, “We don’t receive wisdom; we must discover it for ourselves after a journey that no one can take for us or spare us.”

It’s good to have a friend point us in the right direction and come with us a little way for moral support, but then we need to walk on by ourselves. I hope that this book will be the beginning of your journey, not the end.

The world-famous classical guitarist John Williams was once asked how long it took him to learn to play the guitar. “I’m still learning,” he said.

Audience

This book is designed for those new to Puppet, including system administrators and developers who are looking to manage computer server systems for configuration management. No prior programming or system administration experience is assumed.

Mission

The Puppet 5 Beginner’s Guide gets you up and running with Puppet 5 straight away with real-world examples. It will take you from complete beginner to confident Puppet user with a series of clear examples and case studies. By reading this book, you will learn to manage every aspect of your server setup. You will gain expert understanding of Puppet’s latest and most advanced features and put them to work.

Objectives and achievements

This book helps you to:

  • Install and set up Puppet and discover the latest and most advanced features
  • Automate your IT infrastructure
  • Use the latest features in Puppet 5+ and official modules
  • Manage clouds, containers, and orchestration
  • Learn best practices to make Puppet more reliable and high performing

General structure

  1. Getting started with Puppet
  2. Creating your first manifests
  3. Managing Puppet code with Git
  4. Understanding Puppet resources
  5. Variables, expressions, and facts
  6. Managing data with Hiera
  7. Mastering modules
  8. Classes, roles, and profiles
  9. Managing files with templates
  10. Controlling containers
  11. Orchestrating cloud instances
  12. Putting it all together

Detailed outline

About this book

A brief introduction to the aims and methods of the book, and how to use it.

1. Getting started with Puppet

In this chapter you’ll learn about some of the challenges of managing configuration on servers, some common solutions to this problem, and how automation tools such as Puppet can help. You’ll also learn how to download the GitHub repository containing all the source code and examples in this book, how to set up your own Vagrant virtual machine to run the code, and how to download and install Puppet.

  • Why do we need Puppet anyway?
    • Keeping configuration synchronized
    • Repeating changes across many servers
    • Self-updating documentation
    • Why not just write shell scripts?
    • Why not just use containers?
    • Why not just use serverless?
    • Configuration management tools
  • What is Puppet?
    • Resources and attributes
    • Puppet architectures
  • Getting ready for Puppet
    • Getting the Git repo
    • Installing Virtualbox and Vagrant
    • Running your Vagrant VM
    • Alternative Vagrant VMs
    • Troubleshooting Vagrant

2. Creating your first manifests

The Puppet manifest is the complete expression, in code form, of how your server should be configured. It consists of a set of resources, each of which models some aspect of system configuration, such as a file, or a software package.

In this chapter you’ll learn how to write your first manifest with Puppet, and how to put Puppet to work configuring a server. You’ll also understand how Puppet reads and applies a manifest. You’ll see how to use Puppet to manage the contents of files, how to install packages, and how to control services.

  • Hello, Puppet: your first Puppet manifest
    • Understanding the code
    • Modifying existing files
    • Dry-running Puppet
    • How Puppet applies the manifest
    • Creating a file of your own
  • Managing packages
    • How Puppet applies the manifest
    • Exercise
    • Querying resources with puppet resource
  • Services
    • Getting help on resources with puppet describe
    • The package-file-service pattern
    • Understanding the code
    • Notifying a linked resource
    • Resource ordering with require

3. Managing Puppet code with Git

In this chapter we’ll install Git, create a repo, and import the manifests created in previous chapters. Readers will learn to make some changes to the manifests and commit them to the repo. Readers will learn how to check the revision history of the repo. We will clone our existing repo to a master repo, and deploy it on a new server. We will set up a script and cron job to pull changes from the master repo and apply them automatically.

  • What is version control?
    • Tracking changes
    • Sharing code
  • Creating a Git repo
    • Making your first commit
    • How often should I commit?
    • Branching
  • Distributing Puppet manifests
    • Creating a GitHub account and project
    • Pushing your repo to GitHub
    • Cloning the repo
  • Fetching and applying changes automatically
    • Writing a manifest to set up regular Puppet runs
    • Applying the run-puppet manifest
    • The run-puppet script
    • Testing automatic Puppet runs
    • Managing multiple nodes

4. Understanding Puppet resources

We’ve already met three important types of Puppet resource: packages, files, and services. In this chapter we’ll go into a little more detail on these to see how to exploit their power to the full. We’ll also learn about three other important resource types: user, cron, and exec resources. Along the way we’ll talk about:

  • Files
    • The path attribute
    • Managing whole files
    • Ownership
    • Permissions
    • Directories
    • Trees of files
    • Symbolic links
  • Packages
    • Uninstalling packages
    • Installing specific versions
    • Installing the latest version
    • Installing Ruby gems
    • Installing gems in Puppet’s context
    • Using ensure_packages
  • Services
    • The hasstatus attribute
    • The pattern attribute
    • The hasrestart and restart attributes
  • Users
    • Creating users
    • The user resource
    • The group resource
    • Managing SSH keys
    • Removing users
  • Cron resources
    • Attributes of the cron resource
    • Randomizing cron jobs
    • Removing cron jobs
  • Exec resources
    • Automating manual interaction
    • Attributes of the exec resource
    • The user attribute
    • The onlyif and unless attributes
    • The refreshonly attribute
    • The logoutput attribute
    • The timeout attribute
    • How not to misuse exec resources

5. Variables, expressions, and facts

In this chapter you will learn about variables in Puppet, and discover the different types of data structures and data types that Puppet provides. You will find out how to use variables and values in expressions, and how to apply different resources depending on the value of expressions, using conditional statements. You will also learn how Puppet manifests can get data about the server using Facter, find out which are the most important standard facts, and see how to create your own custom facts. Finally, you will see how to use the each function to loop over arrays and hashes.

  • Introducing variables
    • Using Booleans
    • Interpolating variables in strings
    • Creating arrays
    • Declaring arrays of resources
    • Understanding hashes
    • Setting resource attributes from a hash
  • Introducing expressions
    • Meeting Puppet’s comparison operators
    • Introducing regular expressions
    • Using conditional expressions
    • Making decisions with if statements
    • Choosing options with case statements
  • Finding out facts
    • Using the facts hash
    • Running the facter command
    • Accessing hashes of facts
    • Referencing facts in expressions
    • Using memory facts
    • Discovering networking facts
    • Providing external facts
    • Creating executable facts
  • Iterating over arrays
    • Using the each function
    • Iterating over hashes

6. Managing data with Hiera

In this chapter you will learn why it’s useful to separate program logic from configuration data. You will see how to use Puppet’s built-in Hiera mechanism to store and query config data, including encrypted secrets such as passwords and SSL keys. You will also learn how to use Hiera data to construct Puppet resources directly.

  • Why Hiera?
    • Data needs to be maintained
    • Settings depend on nodes
    • Operating systems differ
    • The Hiera way
  • Setting up Hiera
  • Adding Hiera data to your Puppet repo
    • Troubleshooting Hiera
  • Querying Hiera
    • Typed lookups
  • Types of Hiera data
    • Single values
    • Boolean values
    • Arrays
    • Hashes
  • Interpolation in Hiera data
    • Using lookup()
    • Using alias()
    • Using literal()
  • The hierarchy
    • Dealing with multiple values
    • Merge behaviours
    • Data sources based on facts
    • What belongs in Hiera?
  • Creating resources with Hiera data
    • Building resources from Hiera arrays
    • Building resources from Hiera hashes
    • The advantages of managing resources with Hiera data
  • Managing secret data
    • Setting up GnuPG
    • Adding an encrypted Hiera source
    • Creating an encrypted secret
    • How Hiera decrypts secrets
    • Editing or adding encrypted secrets
    • Distributing the decryption key

7. Mastering modules

In this chapter you will learn about Puppet Forge, the public repository for Puppet modules, and you’ll see how to install and use third-party modules from Puppet Forge with the standard module management tool r10k. You’ll work through examples of using three key Forge modules: puppetlabs-apache, puppetlabs-mysql, and puppet-archive. Finally, you’ll learn how to develop your own Puppet module from scratch, working through a complete example, how to add appropriate metadata for your module, and how to upload it to Puppet Forge.

  • Using Puppet Forge modules
    • What is the Puppet Forge?
    • Finding the module you need
    • Using r10k
    • Understanding the Puppetfile
    • Managing dependencies with generate-puppetfile
  • Using modules in your manifests
    • Using puppetlabs/mysql
    • Using puppetlabs/apache
    • Using puppet/archive
  • Exploring the standard library
    • Safely installing packages with ensure_packages
    • Modifying files in place with file_line
    • Introducing some other useful functions
    • The pry debugger
  • Writing your own modules
    • Creating a repo for your module
    • Writing the module code
    • Creating and validating the module metadata
    • Tagging your module
    • Installing your module
    • Applying your module
    • More complex modules
    • Uploading modules to the Puppet Forge

8. Classes, roles, and profiles

In this chapter you will explore the details of Puppet classes, the distinction between defining a class and creating instances of that class, and how to pass parameters to classes. You’ll learn how to create defined resource types, and how they differ from classes. You’ll also see how to organize your Puppet code using the concepts of roles, and profiles, and how to use Hiera data to assign different classes to nodes.

  • Classes
    • The class keyword
    • Declaring parameters to classes
    • Automatic parameter lookup from Hiera data
  • Parameter data types
    • Available data types
    • Content type parameters
    • Range parameters
    • Flexible data types
  • Defined resource types
  • Type aliases
  • Managing classes with Hiera
    • Using include with lookup()
    • Common and per-node classes
  • Roles and profiles
    • Roles
    • Profiles

9. Managing files with templates

In this chapter we’ll learn about an important and powerful feature of Puppet: the template. We’ll see how to use a simple template to interpolate the values of Puppet variables, facts, and Hiera data into a file, and we’ll also introduce more complex templates using iteration and conditional statements to generate dynamic configuration files.

  • What are templates?
    • The dynamic data problem
    • Puppet template syntax
  • Using templates in your manifests
    • Referencing template files
    • Inline templates
    • Template tags
    • Computations in templates
    • Conditional statements in templates
  • Iteration in templates
    • Iterating over Facter data
    • Iterating over structured facts
    • Iterating over Hiera data
  • Working with templates
    • Passing parameters to templates
    • Validating template syntax
    • Rendering templates on the command line
  • Legacy ERB templates

10. Controlling containers

In this chapter we’ll look at the emerging topic of containers and see how it relates to configuration management. We’ll see how to use Puppet to manage Docker itself, as well as images and containers, and explore some different strategies for managing configuration within containers.

  • Understanding containers
    • The deployment problem
    • Options for deployment
    • Introducing the container
    • What Docker does for containers
    • Deployment with Docker
    • Building Docker containers
    • The layered filesystem
    • Managing containers with Puppet
  • Managing Docker with Puppet
    • Installing Docker
    • Running a Docker container
    • Stopping a container
    • Running multiple instances of a container
  • Managing Docker images
    • Building images from Dockerfiles
    • Managing Dockerfiles
  • Building dynamic containers
    • Configuring containers with templates
    • Self-configuring containers
  • Persistent storage for containers
    • Host-mounted volumes
    • Docker volumes
  • Networking and orchestration
    • Connecting containers
    • Container orchestration
    • What is orchestration?
    • What orchestration tools are available?
  • Running Puppet inside containers
    • Are containers mini VMs or single processes?
    • Configuring containers with Puppet
    • Containers need Puppet too

11. Orchestrating cloud resources

In this chapter we will learn how to use Puppet to create and manage AWS cloud resources, such as EC2 instances.

  • Introducing the cloud
    • Automating cloud provisioning
    • Using CloudFormation
    • Using Terraform
    • Using Puppet
  • Setting up an Amazon AWS account
    • Creating an AWS account
    • Creating an IAM policy
    • Creating an IAM user
    • Storing your AWS credentials
  • Getting ready to use puppetlabs/aws
    • Creating a key pair
    • Installing the puppetlabs/aws module
    • Installing the AWS SDK gem
  • Creating EC2 instances with Puppet
    • Choosing an Amazon Machine Image (AMI)
    • Creating the EC2 instance
    • Accessing your EC2 instance
  • VPCs, subnets, and security groups
    • The ec2_securitygroup resource
    • The ec2_instance resource
  • Managing custom VPCs and subnets
    • Creating an instance in a custom VPC
    • The ec2_vpc resource
    • The ec2_vpc_internet_gateway resource
    • The ec2_vpc_routetable resource
    • The ec2_vpc_subnet resource
    • Other AWS resource types
  • Provisioning AWS resources from Hiera data
    • Iterating over Hiera data to create resources
    • Orchestrator nodes
    • Cleaning up unused resources

12. Putting it all together

In this chapter we will apply all the lessons from previous chapters to see what a complete, working infrastructure looks like. This is a complete worked example which readers can use as the basis of their own Puppet codebase, adapting and expanding it as needed.

  • Getting the demo repo
    • Copying the repo
  • Understanding the demo repo
    • The control repo
    • Module management
    • Classes
    • Roles
    • Profiles
    • Users and access control
    • SSH configuration
    • Sudoers configuration
    • Time zone and clock synchronization
    • Puppet configuration
  • The bootstrap process
  • Adapting the repo for your own use
    • Configuring users
    • Adding per-node data files and role classes
    • Modifying the bootstrap credentials
  • Bootstrapping a new node
    • Bootstrapping a Vagrant VM
    • Bootstrapping physical or cloud nodes
    • Using other distributions and providers
  • The beginning

Author Bio

John Arundel is a devops consultant, which means he helps people build world-class web operations teams and infrastructure, and has fun doing it. He was formerly a senior operations engineer at global telco Verizon, designing resilient, high-performance infrastructures for major corporations like Ford, McDonald’s, and Bank of America. He is now an independent consultant, working closely with selected clients to deliver web-scale performance and enterprise-grade resilience on a startup budget.

He likes writing books, especially about Puppet (The Puppet Cookbook is available from the same publisher). It seems that at least some people enjoy reading them, or maybe they just like the pictures. He also provides training and coaching on Puppet and devops, which it turns out is far harder than simply doing the work himself.

Off the clock, he is a medal-winning competitive rifle and pistol shooter, and a decidedly uncompetitive piano player. He lives in a small cottage in Cornwall, England and believes, like Cicero, that if you have a garden and a library, then you have everything you need.

You may like to follow him on Twitter at @bitfield.

Acknowledgments

My grateful thanks are due to Jo Rhett, who made innumerable improvements and suggestions to this book, and whose Puppet expertise and clarity of writing I can only strive to emulate. Also to the original Puppet master, Luke Kanies, who created a configuration management tool that sucks less, and my many other friends at Puppet. Many of the key ideas in this book came from them and others including Przemyslaw ‘SoboL’ Sobieski, Peter Bleeck, and Igor Galić.

The techniques and examples in the book come largely from real production codebases, of my consulting clients and others, and were developed with the indispensable assistance of my friends and colleagues Jon Larkowski, Justin Domingus, Walter Smith, Ian Shaw, and Mike Thomas. Special thanks are also due to the Perseids Project at Tufts University, and most of all to the inestimable Bridget Almas, who patiently read and tested everything in the book several times and made many valuable suggestions, not to mention providing continuous moral support, love, and guidance throughout the writing process. This book is for her.

Puppet 4.10 Beginner's Guide, 2nd Edition

Order your copy now on Amazon.com (Amazon.co.uk)

A new edition of the bestselling Puppet Beginner’s Guide is now available, covering all the latest features of Puppet 4.10. Learn Puppet from scratch, from installation to managing complex infrastructures, including brand new sections on managing Amazon AWS cloud resources and Docker containers with Puppet. Every code example is available right now online in a special GitHub repo just for this book. There is also a complete, working Puppet infrastructure for you to download, adapt, and use on your own servers.

Praise for the Puppet Beginner’s Guide

Reviews of the previous edition:

Well-written… smoothly-structured… John Arundel’s new book is a solid guide to helping you become both a proficient Puppet user and a more efficient, knowledgeable, and versatile system administrator.”
Slashdot

I can’t imagine having found anything better. This book avoids the stress and boredom of a slow-paced learning session by diving right in to the most important concepts. It was all I needed to feel confident at work, and it only took an hour before I was writing my own code.”
Amazon.com

John Arundel is both a superb engineer and just as importantly an excellent teacher/communicator. Currently the best Puppet book available and essential for anyone wanting to learn Puppet.”
Amazon.com (Kindle edition)

Audience

This book is designed for those new to Puppet, including system administrators and developers who are looking to manage computer server systems for configuration management. No prior programming or system administration experience is assumed.

Mission

The Puppet Beginner’s Guide gets you up and running with Puppet straight away with real-world examples. It will take you from complete beginner to confident Puppet user with a series of clear examples and case studies. By reading this book, you will learn to manage every aspect of your server setup. You will gain expert understanding of Puppet’s latest and most advanced features and put them to work.

Objectives and achievements

This book helps you to:

  • Install and set up Puppet and discover the latest and most advanced features
  • Automate your IT infrastructure
  • Use the latest features in Puppet 4+ and official modules
  • Manage clouds, containers, and orchestration
  • Learn best practices to make Puppet more reliable and high performing

General structure

  1. Getting started with Puppet
  2. Creating your first manifests
  3. Managing Puppet code with Git
  4. Understanding Puppet resources
  5. Variables, expressions, and facts
  6. Managing data with Hiera
  7. Mastering modules
  8. Classes, roles, and profiles
  9. Managing files with templates
  10. Controlling containers
  11. Orchestrating cloud instances
  12. Putting it all together

Detailed outline

About this book

A brief introduction to the aims and methods of the book, and how to use it.

1. Getting started with Puppet

In this chapter you’ll learn about some of the challenges of managing configuration on servers, some common solutions to this problem, and how automation tools such as Puppet can help. You’ll also learn how to download the GitHub repository containing all the source code and examples in this book, how to set up your own Vagrant virtual machine to run the code, and how to download and install Puppet.

  • Why do we need Puppet anyway?
    • Keeping configuration synchronized
    • Repeating changes across many servers
    • Self-updating documentation
    • Why not just write shell scripts?
    • Why not just use containers?
    • Why not just use serverless?
    • Configuration management tools
  • What is Puppet?
    • Resources and attributes
    • Puppet architectures
  • Getting ready for Puppet
    • Getting the Git repo
    • Installing Virtualbox and Vagrant
    • Running your Vagrant VM
    • Alternative Vagrant VMs
    • Adding Puppet to your path
    • Troubleshooting Vagrant
    • Installing Puppet manually

2. Creating your first manifests

The Puppet manifest is the complete expression, in code form, of how your server should be configured. It consists of a set of resources, each of which models some aspect of system configuration, such as a file, or a software package.

In this chapter you’ll learn how to write your first manifest with Puppet, and how to put Puppet to work configuring a server. You’ll also understand how Puppet reads and applies a manifest. You’ll see how to use Puppet to manage the contents of files, how to install packages, and how to control services.

  • Hello, Puppet: your first Puppet manifest
    • Understanding the code
    • Modifying existing files
    • Dry-running Puppet
    • How Puppet applies the manifest
    • Creating a file of your own
  • Managing packages
    • How Puppet applies the manifest
    • Exercise
    • Querying resources with puppet resource
  • Services
    • Getting help on resources with puppet describe
    • The package-file-service pattern
    • Understanding the code
    • Notifying a linked resource
    • Resource ordering with require

3. Managing Puppet code with Git

In this chapter we’ll install Git, create a repo, and import the manifests created in previous chapters. Readers will learn to make some changes to the manifests and commit them to the repo. Readers will learn how to check the revision history of the repo. We will clone our existing repo to a master repo, and deploy it on a new server. We will set up a script and cron job to pull changes from the master repo and apply them automatically.

  • What is version control?
    • Tracking changes
    • Sharing code
  • Creating a Git repo
    • Making your first commit
    • How often should I commit?
    • Branching
  • Distributing Puppet manifests
    • Creating a GitHub account and project
    • Pushing your repo to GitHub
    • Cloning the repo
  • Fetching and applying changes automatically
    • Writing a manifest to set up regular Puppet runs
    • Applying the run-puppet manifest
    • The run-puppet script
    • Testing automatic Puppet runs
    • Managing multiple nodes

4. Understanding Puppet resources

We’ve already met three important types of Puppet resource: packages, files, and services. In this chapter we’ll go into a little more detail on these to see how to exploit their power to the full. We’ll also learn about three other important resource types: user, cron, and exec resources. Along the way we’ll talk about:

  • Files
    • The path attribute
    • Managing whole files
    • Ownership
    • Permissions
    • Directories
    • Trees of files
    • Symbolic links
  • Packages
    • Uninstalling packages
    • Installing specific versions
    • Installing the latest version
    • Installing Ruby gems
    • Installing gems in Puppet’s context
    • Using ensure_packages
  • Services
    • The hasstatus attribute
    • The pattern attribute
    • The hasrestart and restart attributes
  • Users
    • Creating users
    • The user resource
    • The group resource
    • Managing SSH keys
    • Removing users
  • Cron resources
    • Attributes of the cron resource
    • Randomizing cron jobs
    • Removing cron jobs
  • Exec resources
    • Automating manual interaction
    • Attributes of the exec resource
    • The user attribute
    • The onlyif and unless attributes
    • The refreshonly attribute
    • The logoutput attribute
    • The timeout attribute
    • How not to misuse exec resources

5. Variables, expressions, and facts

In this chapter you will learn about variables in Puppet, and discover the different types of data structures and data types that Puppet provides. You will find out how to use variables and values in expressions, and how to apply different resources depending on the value of expressions, using conditional statements. You will also learn how Puppet manifests can get data about the server using Facter, find out which are the most important standard facts, and see how to create your own custom facts. Finally, you will see how to use the each function to loop over arrays and hashes.

  • Introducing variables
    • Using Booleans
    • Interpolating variables in strings
    • Creating arrays
    • Declaring arrays of resources
    • Understanding hashes
    • Setting resource attributes from a hash
  • Introducing expressions
    • Meeting Puppet’s comparison operators
    • Introducing regular expressions
    • Using conditional expressions
    • Making decisions with if statements
    • Choosing options with case statements
  • Finding out facts
    • Using the facts hash
    • Running the facter command
    • Accessing hashes of facts
    • Referencing facts in expressions
    • Using memory facts
    • Discovering networking facts
    • Providing sxternal facts
    • Creating executable facts
  • Iterating over arrays
    • Using the each function
    • Iterating over hashes

6. Managing data with Hiera

In this chapter you will learn why it’s useful to separate program logic from configuration data. You will see how to use Puppet’s built-in Hiera mechanism to store and query config data, including encrypted secrets such as passwords and SSL keys. You will also learn how to use Hiera data to construct Puppet resources directly.

  • Why Hiera?
    • Data needs to be maintained
    • Settings depend on nodes
    • Operating systems differ
    • The Hiera way
  • Setting up Hiera
  • Adding Hiera data to your Puppet repo
    • Troubleshooting Hiera
  • Querying Hiera
    • Typed lookups
  • Writing Hiera data
    • File header
    • Single values
    • Boolean values
    • Arrays
    • Hashes
    • Interpolation
  • The hierarchy
    • Dealing with multiple values
    • Merge behaviours
    • Data sources based on facts
    • What belongs in Hiera?
  • Creating resources with Hiera data
    • Building resources from Hiera arrays
    • Building resources from Hiera hashes
    • The advantages of managing resources with Hiera data
  • Managing secret data
    • Setting up GnuPG
    • Setting up hiera-eyaml-gpg
    • Creating an encrypted secret
    • How Hiera decrypts secrets
    • Editing or adding encrypted secrets
    • Distributing the decryption key

7. Mastering modules

In this chapter you will learn about Puppet Forge, the public repository for Puppet modules, and you’ll see how to install and use third-party modules from Puppet Forge, using three of the most popular module management tools: puppet-module, librarian-puppet, and R10K. You’ll work through examples of using three key Forge modules: puppetlabs-apache, puppetlabs-mysql, and puppet-archive. Finally, you’ll learn how to develop your own Puppet module from scratch, working through a complete example, how to add appropriate metadata for your module, and how to upload it to Puppet Forge.

  • Using Puppet Forge modules
    • What is the Puppet Forge?
    • Finding the module you need
    • Using r10k
    • Understanding the Puppetfile
    • Managing dependencies with generate-puppetfile
  • Using modules in your manifests
    • Using puppetlabs/mysql
    • Using puppetlabs/apache
    • Using puppet/archive
  • Exploring the standard library
    • Safely installing packages with ensure_packages
    • Modifying files in place with file_line
    • Introducing some other useful functions
    • The pry debugger
  • Writing your own modules
    • Creating a repo for your module
    • Writing the module code
    • Creating and validating the module metadata
    • Tagging your module
    • Installing your module
    • Applying your module
    • More complex modules
    • Uploading modules to the Puppet Forge

8. Classes, roles, and profiles

In this chapter you will explore the details of Puppet classes, the distinction between defining a class and creating instances of that class, and how to pass parameters to classes. You’ll learn how to create defined resource types, and how they differ from classes. You’ll also see how to organize your Puppet code using the concepts of nodes, roles, and profiles.

  • Classes
    • The class keyword
    • Declaring parameters to classes
    • Automatic parameter lookup from Hiera data
  • Parameter data types
    • Available data types
    • Range parameters
    • Content type parameters
    • Flexible data types
  • Defined resource types
  • Node definitions, roles, and profiles
    • Nodes
    • Roles
    • Profiles

9. Managing files with templates

In this chapter we’ll learn about an important and powerful feature of Puppet: the template. We’ll see how to use a simple template to interpolate the values of Puppet variables, facts, and Hiera data into a file, and we’ll also introduce more complex templates using iteration and conditional statements to generate dynamic configuration files.

  • What are templates?
    • The dynamic data problem
    • Puppet template syntax
  • Using templates in your manifests
    • Referencing template files
    • Inline templates
    • Template tags
    • Computations in templates
    • Conditional statements in templates
  • Iteration in templates
    • Iterating over Facter data
    • Iterating over structured facts
    • Iterating over Hiera data
  • Working with templates
    • Passing parameters to templates
    • Validating template syntax
    • Rendering templates on the command line
  • Legacy ERB templates

10. Controlling containers

In this chapter we’ll look at the emerging topic of containers and see how it relates to configuration management. We’ll see how to use Puppet to manage Docker itself, as well as images and containers, and explore some different strategies for managing configuration within containers.

  • Understanding containers
    • The deployment problem
    • Options for deployment
    • Introducing the container
    • What Docker does for containers
    • Deployment with Docker
    • Managing containers with Puppet
  • Managing Docker with Puppet
    • Installing Docker
    • Running a Docker container
    • Stopping a container
    • Running multiple instances of a container
  • Managing Docker images
    • Building images from Dockerfiles
    • Managing Dockerfiles
  • Building dynamic containers
    • Configuring containers with templates
    • Self-configuring containers
  • Persistent storage for containers
    • Host-mounted volumes
    • Docker volumes
  • Networking and orchestration
    • Connecting containers
    • Container orchestration
    • What is orchestration?
    • What orchestration tools are available?
  • Running Puppet inside containers
    • Are containers mini-VMs or single processes?
    • Configuring containers
    • Containers need Puppet too

11. Orchestrating cloud resources

In this chapter we will learn how to use Puppet to create and manage AWS cloud resources, such as EC2 instances.

  • Introducing the cloud
    • Automating cloud provisioning
    • CloudFormation
    • Terraform
  • Managing AWS cloud resources with Puppet
    • Setting up an Amazon AWS account
    • Generating AWS access keys
    • Installing the AWS SDK and credentials
    • Creating a key pair
    • Installing the puppetlabs/aws module
    • Choosing an Amazon Machine Image (AMI)
    • Creating an EC2 instance
    • Accessing your EC2 instance
    • The ec2_securitygroup resource
    • The ec2_instance resource
    • Creating a VPC and subnet
    • The ec2_vpc resource
    • The ec2_vpc_internet_gateway resource
    • The ec2_vpc_routetable resource
    • The ec2_vpc_subnet resource
    • Provisioning AWS resources from Hiera data
    • Cleaning up
    • Other AWS resources
    • Orchestrator nodes
  • Other cloud providers
    • Google Compute Engine
    • Microsoft Azure
    • DigitalOcean

12. Putting it all together

In this chapter we will apply all the lessons from previous chapters to see what a complete, working infrastructure looks like. This is a complete worked example which readers can use as the basis of their own Puppet codebase, adapting and expanding it as needed.

  • Getting the demo repo
  • Understanding the demo repo
    • The control repo
    • Module management
    • Nodes, roles, and profiles
    • Users and access control
    • SSH configuration
    • Sudoers configuration
    • Timezone and clock synchronization
    • Puppet configuration
  • The bootstrap process
  • Adapting the repo for your own use
    • Copying the repo
    • Configuring users
    • Adding node and role declarations
    • Modifying the bootstrap credentials
    • Bootstrapping a new node

Author Bio

John Arundel is a devops consultant, which means he helps people build world-class web operations teams and infrastructure, and has fun doing it. He was formerly a senior operations engineer at global telco Verizon, designing resilient, high-performance infrastructures for major corporations like Ford, McDonald’s, and Bank of America. He is now an independent consultant, working closely with selected clients to deliver web-scale performance and enterprise-grade resilience on a startup budget.

He likes writing books, especially about Puppet (The Puppet Cookbook is available from the same publisher). It seems that at least some people enjoy reading them, or maybe they just like the pictures. He also provides training and coaching on Puppet and devops, which it turns out is far harder than simply doing the work himself.

Off the clock, he is a medal-winning competitive rifle and pistol shooter, and a decidedly uncompetitive piano player. He lives in a small cottage in Cornwall, England and believes, like Cicero, that if you have a garden and a library, then you have everything you need.

You may like to follow him on Twitter at @bitfield.

My grateful thanks are due to Jo Rhett, who made innumerable improvements and suggestions to this book, and whose Puppet expertise and clarity of writing I can only strive to emulate. Also to the original Puppet master, Luke Kanies, who created a configuration management tool that sucks less, and my many other friends at Puppet. Many of the key ideas in this book came from them and others including Przemyslaw ‘SoboL’ Sobieski, Peter Bleeck, and Igor Galić

The techniques and examples in the book come largely from real production codebases, of my consulting clients and others, and were developed with the indispensable assistance of my friends and colleagues Jon Larkowski, Justin Domingus, Walter Smith, Ian Shaw, and Mike Thomas. Special thanks are also due to the Perseids Project at Tufts University, and most of all to the inestimable Bridget Almas, who patiently read and tested everything in the book several times and made many valuable suggestions, not to mention providing continuous moral support, love, and guidance throughout the writing process. This book is for her.

The Puppet 3 Cookbook

Build reliable, scalable, secure, and high-performance systems to fully utilize the power of cloud computing.

Over 10,000 copies sold

Buy Puppet 3 Cookbook (Kindle edition) now on Amazon.co.uk or Amazon.com

Puppet 4 update: While most of the techniques and code examples in the Puppet 3 Cookbook are still valid for Puppet 4, there is a brand new book available which is up to date with the very latest Puppet version: the Puppet 4.10 Beginner’s Guide.

Covers Puppet 3

The Puppet 3 Cookbook is fully updated to cover Puppet 3 and with masses of new material, tips, recipes, ideas, and fully-working example code to help you kick-start your Puppet infrastructure. If you enjoyed the Puppet 3 Beginners Guide, or if you’re an experienced Puppet user already, the Puppet 3 Cookbook is the book to take your Puppet knowledge to the next level.

  • Shows you how to use Puppet 3 to take control of your servers and desktops, with detailed step-by-step instructions

  • Written in a simple, practical style by a professional systems administrator and Puppet expert, every recipe has detailed step-by-step instructions showing you the exact commands and configuration settings you need.

  • Packed with tips and inspiring ideas for using Puppet to automate server builds, deployments, and workflows.

  • Covers all the popular tools and frameworks used with Puppet: Hiera, RSpec, puppet-lint, and more

  • Teaches you how to extend Puppet with custom functions, types, and providers.

What they said

“A mandatory read for anyone using Puppet. It is a guide I use almost daily.”
“The book that I wanted ‘Pro Puppet’ to be.”
“Very good insights… really has value. Well written, easy and fun to read.”
“Excellent book even for experienced Puppet practioners. I wanted to enhance my Puppet knowledge and this book does just that. From using tags to encrypting sensitive data or with HieraGPG or creating your own custom function this book covers a tremendous amount of material.”
“This book has some excellent examples and a good overview of some of the more advanced features of Puppet.”
“Practical, handy, and useful… well organized.”
“Great reference book.”
“Excellent… I definitely recommend this book.”

What’s in it

The book takes the reader from a basic knowledge of Puppet to a complete and expert understanding of Puppet’s latest and most advanced features, community best practices, writing great manifests, scaling and performance, and how to extend Puppet by adding your own providers and resources. It starts with help on how to set up and expand your Puppet infrastructure, progresses through detailed information on the language and features, external tools, reporting, monitoring, and troubleshooting, and concludes with many specific recipes for managing popular applications.

The book includes real examples from production systems and techniques that are in use in some of the world’s largest Puppet installations, including a distributed Puppet architecture based on the Git version control system. It covers common problems and errors and shows you how to troubleshoot your Puppet manifests. You’ll be introduced to powerful tools that work with Puppet such as Hiera. You’ll learn how to use objection-orientation and classes to write powerful, reusable manifests, and how to embed Ruby code in templates. You’ll find out how to extend Puppet with custom resource types and providers. The book also explains managing Ruby applications and MySQL databases, building web servers, load balancers, high-availability systems with Heartbeat, and many other state-of-the-art techniques.

More and more systems administration and IT jobs require some knowledge of configuration management, and specifically Puppet. The Puppet 3 Cookbook not only gives you everything you need to become a Puppet expert, but includes powerful code samples and techniques developed over many years of production experience. With it, you’ll save time and effort by automating tedious manual processes, impress your boss by delivering better business value from IT, and future-proof your career by getting to grips with the new technologies revolutionizing the industry.

What you’ll learn

  • Installing and setting up Puppet for the first time
  • Producing eye-catching reports and information for management
  • Understanding common error messages and troubleshooting common problems
  • Managing large networks
  • Taking control of configuration data with Hiera and encrypting secrets with GnuPG
  • Producing reliable, clean, maintainable code to community standards with puppet-lint and rspec-puppet
  • Using classes and inheritance to write powerful Puppet code
  • Deploying configuration files and templates for lightning-fast installations
  • Using virtual machines to build test and staging environments, and production systems on cloud platforms such as EC2
  • Automating every aspect of your systems including provisioning, deployment and change management
  • Making Puppet reliable, performant, and scalable

Who the book is for

The book is for anyone who builds and administers servers, especially in a web operations context. It requires some experience of Linux systems administration, including familiarity with the command line, file system, and text editing. No programming experience is required.

Buy the Puppet 3 Cookbook

Also available

The Puppet 3 Beginner’s Guide

Puppet 3 Beginner's Guide - out now

I’ve looked at lots of different tutorials on Puppet. Yours have been by far the most practical and easiest to understand.”

Those nice people at Packt Publishing have asked me to write a follow-up to the surprisingly successful Puppet Cookbook. This book, the Puppet 3 Beginner’s Guide caters for those who are completely new to Puppet, and takes you through installation, setup, writing your first manifests, using Git to manage and distribute your manifests, managing users, reporting and troubleshooting, and a few other useful things.

Buy the Puppet 3 Beginner’s Guide on Amazon.com (Kindle edition)

Buy the Puppet 3 Beginner’s Guide on Amazon.co.uk (Kindle edition)

Some of the most respected people working in devops and configuration management have contributed to the book’s development by reading early chapters, giving feedback, testing code, and making suggestions.

What they said

Well-written… smoothly-structured… John Arundel’s new book is a solid guide to helping you become both a proficient Puppet user and a more efficient, knowledgeable, and versatile system administrator.”
Slashdot

I can’t imagine having found anything better. This book avoids the stress and boredom of a slow-paced learning session by diving right in to the most important concepts. It was all I needed to feel confident at work, and it only took an hour before I was writing my own code.”
Amazon.com

John Arundel is both a superb engineer and just as importantly an excellent teacher/communicator. Currently the best Puppet book available and essential for anyone wanting to learn Puppet.”
Amazon.com (Kindle edition)

Outline

Introduction to Puppet

What’s the problem with computers?

In this chapter you’ll learn what Puppet is, and what it can help you do. Whether you’re a system administrator, a developer who needs to fix servers from time to time, or just someone who’s annoyed at how long it takes to set up a new laptop, you’ll have come across the kind of problems Puppet is designed to solve.

  • What is configuration management?
  • What problems is Puppet trying to solve?
  • How do you work differently as a systems administrator using Puppet?

First steps with Puppet

In this chapter you’ll learn how to install Puppet, how to write your first manifest, and how to put Puppet to work configuring a server. You’ll also understand how Puppet reads and applies a manifest.

  • Installing Puppet
  • Creating your first manifest
  • Making your first changes on a machine with Puppet

Packages, files, and services

The most common types of resources you’ll manage with Puppet are packages, files, and services. They often occur together, with a package providing a service, and the service requiring a configuration file. In this chapter you’ll see how to use Puppet to manage these resources effectively.

  • Managing packages
  • Managing services
  • Resource dependencies
  • Managing config files
  • The package - file - service pattern
  • A complete worked example using Nginx and a simple website

Managing Puppet with Git

In this chapter you’ll learn how to use the Git version control system to manage your Puppet manifests. I’ll also show you how to use Git to distribute the manifests to multiple machines, so that you can start managing your whole network with Puppet.

  • Introduction to Git
  • Using Git to version control your Puppet code
  • Committing changes
  • Looking at revision history
  • Reverting changes
  • Merging commits
  • Using Git to deploy Puppet manifests to multiple servers
  • Setting up remote access to Git
  • Pulling updates and running Puppet from cron
  • Using feature branches

Managing users

In this chapter you’ll learn how to use Puppet to create and manage user accounts, configure SSH access and keys, and control user privileges via sudo.

  • Creating user accounts with Puppet
  • Configuring SSH
  • Distributing SSH keys
  • Controlling sudo privileges
  • Locking and removing user accounts.

Tasks and templates

In this chapter you’ll learn how to use Puppet’s resource types to run commands, schedule regular tasks, and distribute large trees of files. You’ll also find out how to insert values dynamically into files using templates.

  • Using exec resources to run commands
  • Using cron resources to run scheduled jobs
  • Deploying trees of files
  • Using templates

Definitions and classes

In this chapter you’ll learn how to group resources into reusable clumps that you can refer to by name, making it easy to create lots of similar resources at once. You can also make your Puppet manifests shorter, neater, and more readable by eliminating duplicated code.

  • Arrays
  • Definitions
  • Parameters
  • Classes

Expressions and logic

In this chapter you’ll learn how to make choices in your Puppet manifests, how to do arithmetic, logic, and string operations in the Puppet language, and how to use regular expressions to match patterns in strings. You’ll also find out about some useful Puppet data types: arrays and hashes.

  • If statements
  • Case statements
  • Selectors
  • Operators
  • Regular expressions
  • Capture variables
  • Regular expression substitutions
  • Arrays and hashes
  • Multi-level hashes

Reporting and troubleshooting

In this chapter you’ll learn how to get information on what Puppet’s doing, when it runs, the changes it makes, how to monitor Puppet, and what to do about many common errors you may encounter.

  • Summaries
  • Reports
  • Dry-run mode
  • Debug mode
  • Printing messages
  • Exec output
  • Monitoring Puppet
  • Causes of Puppet run failures
  • Common Puppet errors and how to fix them

Moving on up

In this chapter you’ll learn some simple principles for writing better Puppet manifests, including style and code layout. You’ll also find out what resources are available for learning more about Puppet, and get some ideas for more advanced projects to build your Puppet knowledge and improve your infrastructure.

  • Style guidelines
  • Module design
  • Learning more
  • Project ideas

The Puppet 3 Beginner’s Guide is available on Amazon or via the publisher:

Buy the Puppet 3 Beginner’s Guide on Amazon.com (Kindle edition)

Buy the Puppet 3 Beginner’s Guide on Amazon.co.uk (Kindle edition)

Buy the Puppet 3 Beginner’s Guide from Packt Publishing

Also available

The Puppet 3 Cookbook

Syndicate content