Skip navigation.

Puppet Tutorial for Linux: Powering up with Puppet

Important

This tutorial is rather out of date, and I’m afraid I haven’t had time to update it, because I’ve been working on a whole book that’ll teach you Puppet!

The Puppet 3 Beginners Guide is available now. I recommend you check it out, but I’ll keep the tutorial content here for historical interest, if nothing else.

If you’re a more advanced Puppet user, you might prefer to go straight to The Puppet Cookbook.

Puppet tutorial series

Puppet tutorial part 1: Powering up with Puppet 2.6

This Linux Puppet tutorial will help you install Puppet for the first time and start managing your servers. Server configuration management (CM) is big news in the IT world these days. Rightly so, because Linux automation, devops and CM tools like Puppet and Chef can save you an enormous amount of time and money and help you build a really reliable and automated Linux infrastructure. In this tutorial, I’ll show you how to set up Puppet on Linux.

If you’re a sysadmin, or anyone else who manages a bunch of servers, CM tools can help you create patterns or recipes which you can use to build lots of identical servers, or cloud instances, or re-use in different places and for different applications. Automating Linux servers is a snap with Puppet. Puppet can manage thousands of servers as easily as just one or two - but let’s start with one or two!

If you’re a developer, Linux configuration management lets you write code which describes how servers should be set up - saving you the time and effort of doing it manually, and letting you create large, load-balanced groups of interchangeable servers which are guaranteed to be identically configured.

Installing Puppet

So much for the sales pitch. Let’s take a look at the steps required to get up and running with your first Puppet install (we’ll come to Chef in a later article).

Puppet example

For this Puppet example install, we’re going to assume you’re using an Ubuntu Linux machine. You can use a virtual machine for this, running in VMWare, VirtualBox or Xen if you like, or just a spare Linux box that happens to be lying around the office.

First, we need to install Puppet itself (you’ll need to log in as root or sudo su - on your server):

# apt-get install libopenssl-ruby rdoc libopenssl-ruby1.8 libreadline-ruby1.8 libruby1.8 rdoc1.8 ruby1.8
# wget http://puppetlabs.com/downloads/facter/facter-1.5.8.tar.gz
# wget http://puppetlabs.com/downloads/puppet/puppet-2.6.4.tar.gz
# tar xvzf facter-1.5.8.tar.gz
# cd facter-1.5.8
# ruby install.rb
# cd ..
# tar xvzf puppet-2.6.4.tar.gz
# cd puppet-2.6.4
# ruby install.rb
# puppetd --version
2.6.4

There are gems and packages for Puppet, but for the sake of simplicity we’re installing from source; this also guarantees that you’ll get an up-to-date version, as Puppet is under active development and there are frequent new releases.

Creating a Puppet configuration

Next, we need to give Puppet something to do. The code snippets that tell Puppet how a machine should be configured are called manifests - like a shipping manifest, it tells Puppet what things (packages, users, files and other resources) should be present, and what they should contain.

The first thing to do is create a directory structure to hold your manifests. Puppet itself is not too fussy about how these are laid out, but we’re going to follow the Puppet community’s best practices document and use the recommended layout there. To save time, you can download a (nearly) empty template layout. To use this:

# cd /etc
# wget http://bitfieldconsulting.com/files/powering-up-with-puppet.tar.gz
# tar xvzf powering-up-with-puppet.tar.gz

You should now have an /etc/puppet directory with two subdirectories: modules and manifests. Roughly speaking, modules is where Puppet code lives, and manifests is where we specify which code should be applied to each machine under Puppet’s control.

Starting the Puppetmaster

Puppet comes in two parts: a server (the Puppetmaster) which listens for connections from clients and then tells them what manifest they should have, and a client program, which runs on each machine under Puppet control and connects to the Puppetmaster to get its manifest.

In our example, we’re running the master and client on the same machine, so let’s start the server:

# puppet master --mkusers

Running the Puppet client

With the Puppetmaster running, we can now check that everything is working as it should by running the client:

# puppet agent --test --server=`hostname`
info: Caching catalog for localhost.localdomain
info: Applying configuration version '1256130640'
notice: Finished catalog run in 1.23 seconds

Creating Puppet classes

While technically we have just applied a Puppet manifest to the machine, it’s not a very interesting one, as it does nothing at all. Let’s create a manifest that manages a simple service: NTP (the Network Time Protocol). NTP is a daemon which keeps the machine’s clock synchronised with reference servers on the Internet.

First we’re going to create a class for the NTP service (if you haven’t done any object-oriented programming, a class is just a named chunk of code which we can refer to).

I’ve created a (nearly) empty template for you for the NTP class, so go ahead and edit the file /etc/puppet/modules/ntp/manifests/ntp.pp. Currently it looks like this:

class ntp {
}

Change it to this:

class ntp {

    package { "ntp": 
        ensure => installed 
    }

    service { "ntp":
        ensure => running,
    }
}

And save the file. We’ve now created a Puppet manifest which will manage the NTP daemon (ntp).

If you’re using CentOS, RHEL, or another Red Hat-style Linux distro, the name of the NTP service is ntpd, instead of ntp. Change the service resource accordingly.

Creating Puppet nodes

Because Puppet can manage many machines at once, we still need to tell it to apply the NTP manifest to this machine. To do that, edit the file /etc/puppet/manifests/nodes.pp. You’ll see a template that looks like this:

node myserver {
}

Change it to this:

node myserver {
    include ntp
}

(Replace myserver with the name of your machine - that is, the output of hostname -s.)

When Puppet runs, it looks for a node definition that matches the name of the client machine, and applies all the classes that it finds listed there with include.

You’ve now told Puppet that the server myserver (or whatever your machine is named) should have the NTP manifest applied to it.

Applying changes with Puppet

When you run Puppet again:

# puppet agent --test --server=`hostname`
info: Caching catalog for localhost.localdomain
info: Applying configuration version '1256130643'
notice: //ntp/Service[ntp]/ensure: ensure changed 'stopped' to 'running'
notice: Finished catalog run in 0.94 seconds

What happened here? Puppet looked up the manifest for myserver and found this:

    package { "ntp": 
        ensure => installed 
    }

    service { "ntp":
        ensure => running,
    }

This says that the package ntp should be installed, and the service ntp should be running. Your system may or may not have ntp already installed, but if not, Puppet will install it. Puppet checks to see if the service is started, and if it’s not, it starts the service for you. If you run Puppet again now:

# puppet agent --test --server=`hostname`
info: Caching catalog for localhost.localdomain
info: Applying configuration version '1256130643'
notice: Finished catalog run in 0.66 seconds

you’ll see that Puppet does nothing, because the manifest is satisfied.

Puppet manifests are declarations

Puppet manifests are not a set of instructions, in the way that a shell script or a Ruby program is. They’re a declaration of how the world should be (or that part of it under Puppet’s control, anyway). If reality differs from the manifest, Puppet takes steps to adjust reality.

This means that Puppet isn’t just useful for building machines. It can also check them regularly to make sure that nothing important has changed, and if it has, to correct it. If you stop the NTP service manually and then run Puppet again, you’ll find Puppet restarts it.

Conversely, if you decide you no longer want NTP running on your machines, you can change the word running to stopped, and Puppet will enforce this policy on every machine that includes the NTP manifest (which could be hundreds or even thousands of servers).

Imagine making a change like that to a large production network by logging in to each machine in turn and running the necessary commands manually. Not only would it be tedious and time-consuming, but if you got interrupted halfway through, you might not remember which machines you’d fixed, and your whole network would be in an inconsistent and unknown state.

In our next thrilling episode

You’ve just powered up your system with Puppet! Next time in Puppet Tutorial part 2: Client and Server, we’ll look at how to set up Puppet to control multiple machines. If you’ve got any comments on this Puppet tuto, do let me know!

Puppet links

Puppet books

If you’re excited about Puppet and want to learn more, may I modestly recommend the Puppet 2.7 Cookbook? The Puppet 2.7 Cookbook takes you beyond the basics to explore the full power of Puppet, showing you in detail how to tackle a variety of real-world problems and applications. At every step it shows you exactly what commands you need to type, and includes full code samples for every recipe.

SCM and Sprinkle

Thanks for that short and sweet intro to puppet. What I find really mandatory is that you manage your modules and manifests with a source code control system. Otherwise you really might mess up your whole infrastructure with no way back :-(

Besides puppet and chef, we’re currently evaluating sprinkle (http://github.com/crafterm/sprinkle/,
http://redartisan.com/2008/5/27/sprinkle-intro). I’m not a big fan of a client-server solution, which applies changes “automagically” - we usually trigger rollouts ourselves. I understand that this can be done with puppet and chef, too, but it’s not the intended use.

Re: SCM and Sprinkle

Thanks! Sprinkle looks really interesting, I’ll check that out!

Your comment about using a source code control system is dead right - and this is essential for server configurations even if you’re not using a CM tool like Puppet, Chef or Sprinkle. . You’ve just given me a great idea for the next blog post! :D

Even with version control, it’s not trivial to roll back a system using Puppet - for example, if you add a user in Puppet and then remove it, Puppet won’t remove the user from the server. It might be useful to have a flag in Puppet that says ‘Everything on this box is managed by Puppet! If it’s not in the manifest, remove it!”

But that could cause other interesting problems…

removing things

That’s what

user { “foo”:
….
ensure => absent
 }

is for.

You can leave that clause in place for
a few days and then clear up the recipe,
or alternatively just ‘userdel’ via ‘Command and Control’ layer
(ssh-in-a-for-loop, controltier, fabric, etc.).

Try to get Puppet from a repo if you can. We use EPEL for our Redhat boxes, other third party repos are available.
“If you’re building from source, you’re doin’ it wrong” :)

Looking forward to the next post :)

Thanks, and a suggestion

I appreciate you having put this together, thank you very much.

I suggest you update your puppet link to puppet-0.25.4 since there are some significant certificate issues with a first-time install of puppet-0.25.0.

ensure NTP installed

My CentOS 5.4 didn’t come with ntp.

yum install ntp

I tried to figure out how to ensure=>present, otherwise: yum install … but I couldn’t.

Re: ensure NTP installed

Right! Thanks for letting me know. I’ll update the tutorial to include this bit:

package { "ntp": ensure => installed }

That should get it working for you.

Pupeteer?

I noticed that a lot of people come to this article through searching for “pupeteer linux”. Is everyone just misspelling “puppeteer linux”, or is there something called Pupeteer that I’m not aware of?

re:

Nice post. What kind of blog platform is this? I don’t think its wordpress. Where did you get your template? Is it prebuilt?

Jesse

Re: Blog platform

It’s powered by Drupal and managed entirely by Puppet! I’m a great believer in eating dogfood, in the computing sense that is. Not actually eating dogfood, because that would be rank.

Error message

Hi,

Thanks for the great tutorial. I followed it to the tee using puppet 2.6.2 and I’m getting the error

err: /Stage[main]/Ntp/Service[ntpd]/ensure: change from stopped to running failed: Could not find init script for ‘ntpd’

when I run $ sudo puppetd —test —server=myhostname

All of the files in your scaffolding are in /etc/puppet and I’ve made the changes you list. Has something changed in puppet 2.6.2 which would cause this?

Thanks,

Sam

Re: Error message

Hi Sam,

It sounds like on your platform, the “ntpd” service is called something different (maybe “ntp”). This is true on Debian/Ubuntu systems, for example.

Try changing the Service definition from

service { “ntpd”:

to

service { “ntp”:

and see if that works.

alias for name changes

is there a mechanism to alias ntpd to ntp to avoid this type of rework … I could see this being useful to transition migration

yo can use the name attribute

yo can use the name attribute to avoid this kind of rework

re:

Good thing I found this linux Puppet stuff as I wasn’t able to find this information elsewhere. I am bookmarking this for future use.

Tracie

re:

That’s great! I’m glad you found it useful. This Puppet tutorial is consistently one of the most popular pages on my site, so I must be doing something right!

I have personally taken the

I have personally taken the Puppet tutorial and as per my opinion everyone should take such type of coaching who are engaged with this platform.

VMware

How would you administer a large number of virtual machines with Puppet?
 Thanks.

What I tend to do is build

What I tend to do is build the VM, install Puppet, run Puppet to bring in all of the standard configuration (user accounts, keys, sudoers, etc), and then freeze that image. When I spin up VMs, a further run of Puppet will bring in any changes since the image was frozen, but otherwise it’s a very fast start. I use this kind of setup to implement dynamic scaling on EC2, for example.

NPT isn't started, but server seems to respond

Hi,

Starting out, trying to get it working. I have a ubuntu laptop on which I’m running your tutorial:
err: /Stage[main]/Ntp/Service[ntp]/ensure: change from stopped to running failed: Could not find init script for ‘ntp’

Googling it gives this page: https://bugs.launchpad.net/ubuntu/+source/ntp/+bug/604717

Hmm, puppet seems broken. :)
Ubuntu 10.10 x86_a64

I don’t think it’s a Puppet

I don’t think it’s a Puppet problem - this is working for me on Ubuntu 10.04.

Can you run ‘service ntp start’? Or ‘/etc/init.d/ntp start’? That is most likely what Puppet will try to run.

Reply

Yup, I can run the second command there and it says ” * Starting NTP server ntpd —> [ OK ]”.

Hmm, that does sound like

Hmm, that does sound like Puppet is having trouble finding the correct way to start the service.

Try adding:

    provider => "debian",

to the definition of the NTP service. Sometimes Puppet doesn’t detect it correctly so this will force it to use the init scripts. Also, see if it works if you change the name of the service in Puppet to ‘ntpd’.

Awesome, puppet rulz!

Cool. The provider-part worked. Puppet may become a dog one day!

Question no. 2. I have .Net/mono web sites and application servers that I want to be able to build and configure with deployment from builds from git/teamcity’s artifacts, to provision to run on some number of virtual machines (vmware workstation or ec2 or windows azure). They can be run with nginx/mono_fastcgi. The end goal is to be able to go to a client, run a puppet script and get a complete CI/OSS development/git/web server/load balancer/app-server/amqp-rabbitmq/azure environment set up in a few minutes/hours — normally this takes a few days for me to set up. Are there any short cuts I can take to get to this nirivana for a creative developer?

How would you recommend getting started with this endeavour? Should I package everything into .deb-files? Keep a “file” object in puppet or how would I integrate them? And what about Windows - .msi-files or a transaction log with transactional NTFS that can be rolled back/compensated or WinRM or MsDeploy or simply Admin-rights for the puppet service and lots of small tools that it can use to check the computer state such that it can make it ‘right’?

There’s quite a few questions

There’s quite a few questions there :)

Email me at hello@bitfieldconsulting.com and I’ll try and answer them in more detail than I can here.

Your tutorial is very

Your tutorial is very detailed in manner. Thanks a lot for sharing this. My officemate have been wanting to learn more about this Puppet thingy but never got the chance to search extensively due to time constraints. I appreciate the links you included. It wraps up everything comprehensively.

Wonderful tutorial. You make

Wonderful tutorial. You make it so simple and easy to understand. Now i can solve someone’s doubt also. It has very good content. Keep helping.

Not working for 10.04

On a fresh 10.04 install the following problems exist -

1. There is no ‘ruby’ bin installed that can run ‘ruby install.rb’ as ruby 1.8 binarys is called ‘ruby1.8’

2. after running the puppet master command and trying to run puppet agent —test the following error is shown:

err: Could not retrieve catalog from remote server: Connection refused - connect(2)

I tried changing ownership of files in /etc/puppet (example template has undefined user id) and also tried explicitly referring to 127.0.0.1 as the server but always the same result.

Full error is: err: Could not

Full error is:

err: Could not retrieve catalog from remote server: Connection refused - connect(2)
warning: Not using cache on failed catalog
err: Could not retrieve catalog; skipping run

I have the same issue. Have

I have the same issue. Have you found any solutions?

On the puppetmaster restart

On the puppetmaster restart the /etc/init.d/puppetmaster restart fixed the problem for me :-D

On the puppetmaster server

On the puppetmaster server restart the service “/etc/init.d/puppetmaster restart” this fixed the problem of connection refused - connect(2) for me.

Puppet

I have downloaded the puppet vm from puppetlabs.com. On running puppet agent —test —server=’learn.puppet.demo’ it also says Connection refused -connect(2) warning: Not using cache on failed catalog
err: could not retrieve catalog etc

puppet agent --test --server=`hostname` ERRORS OUT

I am trying evaluate ‘puppet’ and I am running inot an error.
When I run the command;
puppet agent —test —server=`hostname`

I get the following error;

err: Could not run Puppet configuration client: Could not retrieve local facts: uninitialized constant Facter::IPAddress

I have the same problem with

I have the same problem with you,it is because the suse 11 linux server have the facter and puppet package,so we need to remove the old version and resetup the facter and puppet package and configuration,I have resolve the problem with this method.

# puppet master

# puppet master —mkusers
Could not parse for environment production: Could not find file /root/master.pp

Oh well.

# puppet master

# puppet master —mkusers
Could not parse for environment production: Could not find file /etc/puppet/master.pp

Nor in here!

# puppet agent —test

# puppet agent —test —server=`hostname`
info: Retrieving plugin
err: /File[/var/lib/puppet/lib]: Failed to generate additional resources using ‘eval_generate’: Connection refused - connect(2)
err: /File[/var/lib/puppet/lib]: Could not evaluate: Connection refused - connect(2) Could not retrieve file metadata for puppet://srv5/plugins: Connection refused - connect(2)
err: Failed to prepare catalog: Could not run command from prerun_command: Execution of ‘/etc/puppet/etckeeper-commit-pre’ returned 1:
err: Could not run Puppet configuration client: Could not run command from postrun_command: Execution of ‘/etc/puppet/etckeeper-commit-post’ returned 1:

oops.

A step is missed out: #

A step is missed out:

# puppet master —mkusers

This did not start any deamons, so no puppet server is actaually running, which explains why the next command fails:
puppet agent —test —server=myIPaddress

# ps -eaf|grep puppet
root 17691 12508 0 11:32 pts/6 00:00:00 grep puppet

error

when i gave command puppetd —test in the terminal, I got this output:

info: Retrieving plugin
err: /File[/var/lib/puppet/lib]: Failed to generate additional resources using ‘eval_generate’: Connection refused - connect(2)
err: /File[/var/lib/puppet/lib]: Failed to retrieve current state of resource: Connection refused - connect(2) Could not retrieve file metadata for puppet://puppet/plugins: Connection refused - connect(2)
err: Could not retrieve catalog from remote server: Connection refused - connect(2)
warning: Not using cache on failed catalog
err: Could not retrieve catalog; skipping run

Re:

Try the same as root user (or with sudo) and this should work

re A step is missed out: #

As root run puppet master —mkusers —no-daemonize
And you should see where your errors are in your config files

take the quotes off the

take the quotes off the “ntpd” in /etc/puppet/modules/ntp/manifests/ntp.pp.

That will fix:
err: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not match “ntpd”: at …

error on first test

Hi,

I just follow the installation instruction here up to
# “puppet agent —test —server=`hostname`”
I get the error below

err: Could not retrieve catalog from remote server: Connection refused - connect(2)
warning: Not using cache on failed catalog
err: Could not retrieve catalog; skipping run

of note: I am on Rackspace running Ubuntu 11.10 (Oneiric Ocelot)
Because of the steps above I have 2 python dir on my machine, one says “python” the other “python2.7”
root@noizrecords:/etc# puppet agent —configprint server
returns
 puppet

EC2 is a great for trying out puppet!

Want to learn more about puppet and tryout your modules in a safe environment? Amazon EC2 is a good answer.

Installing a Puppet Master is not a staight forward task and you need to get your puppet master up and running before you can think about using it to configure other clients. The “puppet” profile will load everything that is required to install and configure either a master or a client and loads everything required at boot time (so you get the latest versions). You can create your own AMI or use mine (search for ‘practicalclouds’).

http://www.practicalclouds.com/content/guide/puppet-deployment

The profile installs and configures the following: -

* Puppet Agent or Master, Puppet Dashboard, and Facter
* Apache and Ruby Passenger for performance and scalability.
* MySQL for external resources, the dashboard, and external node classifier.
* MCollective and ActiveMQ for real time orchestration features.
* Puppet Module for downloading modules from the puppet labs module repository.
* Puppet Cloud Provisioner module, Fog, and the AWS Ruby SDK.
* MCollective Plugins for controlling puppet and services, and automatic configuration of puppet environment facts for use in MCollective.
* Inventory and file bucket services for the dashboard.
* Automatic Backups of puppet configs, puppet data, apache configs and MySQL database.

It’s freely available in every AWS Zone and I hope it helps you in getting your platforms up and running.

regards

Dave

Naive question. What

Naive question. What advantages does Puppet give me over pushing configs out over SSH? It’s easy enough to automate the mappings between hostname/IP/template using flat text files, shell utilities and GNU Make. What am I missing that gives Puppet so much Hipster-cred?

Did you get the answer? I am

Did you get the answer? I am wondering the same…

Nope. I’ve asked so-called

Nope. I’ve asked so-called “experts” elsewhere and either got no answer or a bad answer. There’s way too many dubious “technologies” springing up recently to spend any significant time researching. If the problem it solves is not immediately apparent and no one can give me a dumbed down, epiphany-inducing explaination, I’m inclined to believe it’s just hype and marketing.

Also phrased as…

If you can’t explain it to a six year old, you don’t understand it yourself” — Albert Einstein

Guessing you never had to manage someone else's scripts

Caveat: I am a major CFEngine3 proponent over puppet or chef in nearly every use. but for this explanation the technologies are similar enough.

First major difference: everyone on the team is going to speak the same config language. If there are 5 people maintaining your config, it helps to be able to enforce a common language. Otherwise some people are going to prefer bash and some perl. Some are going to create common libraries for their functions, others are not. Hiring decisions are going to be easier.

Second major difference: unless everyone on the team is damned good and probably spend time writing a custom tool, you are going to be using a procedural definition rather than an idempotent declarative configuration. In other words you are going to be saying “copy this file to the target and restart the service” instead of saying “make sure this file exists and contains these lines. Restart the service if you make changes”.

Third major difference: unless everyone on the team is damned good and probably spend time writing a custom tool, you are going to have a problem managing a heterogeneous set of systems. You might say “detect operating system by checking the presence and parsing the text of certain files. If the system is this, then do that; If it is another thing, do this other thing which is similar but not the same as the first; If it is yet another thing, then…” Instead you could simply use built-in classes to determine actions (RHEL use yum, Debian use apt-get to manage packages)

BTW, I would recommend you prepare packages with jenkins, and toss them in a repo rather than compiling on each system. It allows you to more easily change your mind and test the change. It also helps avoid the “crap I have a missing library on sixty of my fifteen hundred systems” problem.

Scripts - effort spent defining how to do stuff
Puppet/Chef - effort spent defining what you want at the end
CFEngine3 - effort spent defining relationships and reasons about what you want at the end.

I recommend reading about promise theory and knowledge management.

“BTW, I would recommend you

BTW, I would recommend you prepare packages with jenkins, and toss them in a repo rather than compiling on each system. It allows you to more easily change your mind and test the change. It also helps avoid the “crap I have a missing library on sixty of my fifteen hundred systems” problem.”

Personally, I would just prefer to write a 20 line spec file, rpmbuild, rpmlint and push it out using good old-fashioned Unix tools. Not exactly sure why I would have to worry about “missing libraries” when RPM has been satisfying dependencies automatically for 15 years now.

This quote immediately jumps to mind:

…Mac users, in the typical hipster fashion, brutally raped the Unix culture, throwing away everything that made it unique because they did not understand it”

I’m gonna go out on a limb here and guess that all Puppet/Chef/CFEngine3 advocates are either selling it….or are Mac users.

Jenkins is as world-changing as CFEngine3/Puppet/Chef

The good:
At least you are using a package management utility. The missing libraries come into play because the post to which I replied mentioned make, which I thought made it look like they compiled their source code on a target system. (Not Good, wasteful and risky)

The maybe bad:
If you prefer to do that without using source control, your change management tracking could be improved.

You describe doing a few things by hand when you notice the source code change, but I would bet you do more. I image that you (using unix tools) put it onto a repository so that it can be retrieved. I bet you do some functional testing. I bet your development is a bit iterative since you might not get things perfectly the first time. You might work for a company that uses more than one Linux Distribution and need to mod your packages to be different for each target platform.

This is all stuff that Jenkins will do for you, and it will do them automatically any time there is a source code change. It is awesome.

All spec files are under

All spec files are under version control. Source code is under version control. All configuration files and host mappings are under version control. GNU Make is what builds the source code before it goes into the packages. The spec files are written once. After that, rpmbuild or a Koji build server takes care of everything according to the spec file.

An upgrade consists of literally bumping a version number, writing a short changelog entry, issuing a build and pushing out the package to staging and then production. The Fedora and RHEL release engineering teams use this exact same workflow. They’ve been doing this for over a decade and now manage more than 22,000 packages.

Custom RPMs can be created, including custom configurations if need be. Hand written scripts also scale just fine. GNU Make is also useful for small automation tasks — not just building source code.

This whole “DevOps” brain damage is just re-inventing old things, poorly.

Koji Build System

Koji and Jenkins appear similar after a cursory look, I just have never used it. It appears that Jenkins has better extensibility through community modules but worse command-line control. That seems a matter of preference and local situation.

I think you may not understand “DevOps” http://www.opscode.com/blog/2010/07/16/what-devops-means-to-me/, but a discussion of an entire philosophy is probably out of scope for this discussion. I certainly don’t think that it is brain damage.

Wait…did you just say

Wait…did you just say “world-changing”?! Maybe you can hook up with this guy and exchange ideas:

http://twitter.com/#!/hipsterhacker

Yes I did

Thank you for the tip, It turns out I am already following that funny account.

Jenkins certainly changed the way I experience the world. Between Jenkins and CM, I am significantly more effective than I was a half-decade ago.

I am not a hipster. I prefer sincerity and awestruck wonder at the shared magic of life. What I understand of hipster culture (admittedly not a lot) is that they prefer detached postmodern cynicism, which I believe is a wasteful way to live.

The fact that I am here is actually a result of that view of life. Some people I follow on twitter were making fun of this discussion, and I decided that rather than laugh at people who did not understand a principal of large systems design, I would explain what I knew of it. In that sharing, we might be able to learn something from one another. Yes it is idealistic and maybe even naive.

http://xkcd.com/1053/

“First major difference:

First major difference: everyone on the team is going to speak the same config language. If there are 5 people maintaining your config, it helps to be able to enforce a common language. Otherwise some people are going to prefer bash and some perl. Some are going to create common libraries for their functions, others are not. Hiring decisions are going to be easier.”

So instead of enforcing, say, Python, you’re now enforcing…. something else….that you just happen to be selling consulting services for? I see what you did there. +1 for smooth marketing.

You did not see what I did there.

Actually you did not see what I did there.

Python is a bad choice for two reasons, one of which it shares with puppet and chef.

Reason 1 - Your configuration management engine depends on an external language interpreter. This is bad, because now your configuration management is subject to the vicissitudes of the developers of the external language interpreter. It will also be bloated and slow.

Reason 2 - Python is a general scripting language. If you do a really great job, you have an application that may be a competitor with the commonly deployed configuration editors, but with a significantly smaller installed user base. But at best, you are going to have something equivalent to Salt, but after having spent a lot of your own time developing something that you could have done with something else for free and quickly.

My economic interest is not in creating work for myself, there are only so many jobs I can take. Clients pay me because I teach them how to operate more complex systems, more reliably, with smaller staff.

Umm, the point was in the

Umm, the point was in the statement, not in the arbitrary choice of Python as an example.

The point

I suspect that we agree that the question is: “Is it better to implement your own configuration management system or invent your own?”

It seems like your preference is to write your own from scratch, with all of the complexity that entails. Mine is to leverage the body of knowledge, work, and troubleshooting done by the community to enable me to do things faster. While I do not understand why you would want to do that and I believe I see significant drawbacks, I wish you the best of luck.

CM approach: You treat the CM

CM approach:

You treat the CM software like a black box and only ask, “what recipe do I need to copy-paste to make X thing happen on Y and Z hosts?” or “which consultant can make this magic happen and how much is he charging?”

Unix approach:

You compose small tools together. The “configuration” also serves as the implementation. Even so, it’s still probably half the size of the equivalent Puppet configuration.

The benefit of the Unix approach is that you can understand it in it’s entirety. The drawback is that you are required to understand it in it’s entirety.

Difference in degree, not type

Thank you for this, I understand the objection now, and basically agree with your statement’s framing.

However, I would add that that the difference is one of degree rather than type.

You use filesystems as a black box because you do not typically have to worry about which inode a part of data is stored on. You use system commands like ‘mv’ as black boxes because you do not worry about which linux kernel system calls are being used to alter the the system. You do not worry about what your encryption key is, or how it is applied to data that will stream through an interface because ssh handles it for you.

I use CM as a black box because I do not worry about what abstractions are taking place to perform my intended operation.

I believe now that I am more comfortable with higher levels of abstraction than you are.

What value does it add to a business operation that you know what is happening a few layers of abstraction down if CM makes that unnecessary?

You make a good point with

You make a good point with that last sentence.

My perspective is that I already have a working system that is basically an imperative (and perhaps less pretty) configuration language rather than general purpose CM software. It does exactly what I need it to do and no more.

The only reason I am interested in Puppet and Chef is to see what I stand to gain by using their declarative language in place of my ad-hoc, imperative language. On first impression it seems more verbose, over-engineered and offering no real payback.

You’re right in saying that I still use the shell utilities as another “black box” though. The difference is that I do understand those utilities at the source level and trust the developers, whereas I have a deep mistrust for system level software written in Ruby (kneejerk maybe).

However, I had honestly never heard of CFEngine before you mentioned it. My first impression of it is much better than Puppet or Chef. Enough to look into it further.

In general what you gain from

In general what you gain from a declarative language versus an imperitive language is an ability for a system to solve its own problems. Essentially the systems configuration problem can be likened to navigation.

The first task in a imperitive program is to determine starting location. To do this you might ask questions like “Are you by the big white house?” or “Are you at the corner of elm and main facing east?”. These answers may actually be misleading, because the person you are giving directions to may be looking at a light gray house or in the wrong city. Then you provide a relative set of instructions “travel forward 1.5 miles, turn left at the light, go three blocks then turn left at the auto parts store, pull into the fourth driveway on the left, then get out of the car and go in the house where there will be cake”. However the directions may not be adequate because the auto parts store might have closed and become a grocery. Maybe there was another driveway put in. Finally your expectation is that at the end, you will have a car in the driveway and a person in your living room expecting cake. However a person might have stolen the car after it was in the driveway, or the dog might have eaten the cake.

In a declarative language you might say:
Living Room has cake on table.
Person exists in Living Room.
Car Exists in driveway at given address.

The person tries to get to the living room. If there is a detour, the car still tries to get to the driveway. The living room will try to produce a cake on a table.

Further CM, continues to ensure correctness. Personally, this is the thing I love about it. If a dog eats the cake, the living room will produce one on a table. If a user performs a privilege escalation and turns off SElinux in violation of Security Policy, then it is reapplied (computer immunology concepts from Mark Burgess) and notice is sent out.

I too share your hesitation about a CM system depending on externally provided language interpreters. I will explain later when I am out of a meeting.

CM that depends on Ruby

I agree with your distrust of using any CM tool that has a dependency on an external language interpreter.

I was talking with Luke Kanies at LISA ‘11 and mentioned that while I wanted to deploy puppet for the resume fodder, I never have been hired at a place where it was the best choice. He asked why I thought that.

I told him that benchmark studies had shown significantly reduced speed of execution and huge memory consumption compared to CFEngine3. In addition the dependency on Ruby made implementation in shops with legacy systems much more difficult. He told me that the memory bloat and performance issues were usually caused by people running an out-of-date version of Ruby, and that improvements were being constantly made, but that people had to make sure that they were using an up to date version of Ruby. He acknoledged that implementations on legacy gear was problematic. I am pretty sure he told me that he chose Ruby after hearing how easy it was to develop in.

I believe that Puppet performance problems being traced to an external dependency problem shows a fundamental flaw in architecture. In my mind my CM package should be unaffected by the integrity and quality of nearly anything else on the system. I also wonder what happens when the service I am trying to provide needs one version of Ruby and my CM product performs optimally with another (I have never solved it because I have not needed to. Maybe they have a way to deal with that.).

I think that puppet is probably best in shops that use Ruby for development, have modern systems, and have consistently good connections between the puppet master and the puppets. Sadly I have never worked there.

Trollin’ for free consultin’…

Trollin’ for free consultin’… How am I doing?

Also, what do you think of this: https://juju.ubuntu.com/ ?

By the way, I think what we

By the way, I think what we are debating here is ultimately this:

http://en.wikipedia.org/wiki/Programming_in_the_large_and_programming_in…

I don’t understand why people

I don’t understand why people always fail to mention Bcfg2 in these discussions. Your second reason is invalid as not only is Bcfg2 “free and quick”. It has also been around slightly longer than Puppet and much longer than Chef.

Arguments against CFEngine3/Puppet/Chef

I wanted to summarize what I believe the objection to CFEngine3/Puppet/Chef is:

But Will, I don’t think I need this.
1. We use configuration management tools to manage source code and configuration files.
2. We use a tool to automatically build and test packages, then place them on a server where they are shared.
3. ( We use a tool we wrote to command each system to be correct across many configurable dimensions in an idempotent way using a declarative syntax, and ((our systems are homogenous)|(we have conditions in the tool to determine the classes of systems we manage))
4. The tool is simple enough that we do not have problems getting new hires up to speed on using and extending it.
5. It is simple to extend for new tasks we are assigned to manage.”
6. We do not regularly experience bugs/problems because the tool is well written.

My response:
“It appears that what you say you have already build is an excellent system for managing change across large numbers of systems. It might be worth releasing into the community so that others can benefit from the massive amount of time you spent and/or your extrodinary talent. In addition, you may find that the community is able to find small bugs you might have missed.

If it turns out that the code maintenance on your custom written tool starts taking too long or you are finding shortcomings that are difficult to solve, you may want to look at the tools that are used by thousands of large companies and that have been developed by a whole lot of full time developers. It might turn out that they solved the problem you are having many years ago. I know it happened to me once when I found out a tool I spent far too long writing simply wasn’t good enough for what I wanted to do.”

Altering configuration files using puppet

Thanks for the article, however, I am wondering how to efficiently manage/edit the configuration files of the services. Example If I want to change parameters in ntp config file how I should be doing so. An example will be of great help.

> package { “ntp”: ensure =>

> package { “ntp”: ensure => installed }

Fedora/RedHat KickStart:

%packages
ntp
 %end

RPM meta-package:

Requires: ntp

So….. Exactly what does this Puppet junk do again?

Missing elements in tutorial

I had to make a lot of changes that aren’t in the tutorial.

Firstly, /etc/hosts also has to have puppetd, and puppet-master for localhost. The user ‘puppet’ is client only.

Secondly, I had to add the following lines to /etc/sysconfig/iptables and restart the firewall:

-A INPUT -m state —state NEW -m tcp -p tcp —dport 443 -j ACCEPT
-A INPUT -m state —state NEW -m tcp -p tcp —dport 1110 -j ACCEPT
-A INPUT -m state —state NEW -m tcp -p tcp —dport 8140 -j ACCEPT
-A INPUT -m state —state NEW -m tcp -p tcp —dport 61613 -j ACCEPT

Thirdly, on modern CentOS distros, it is “ntpd” not “ntp”.

For what it’s worth…

Thanks a lot for this

Thanks a lot for this wonderful article. Our team here is looking to see whether if this is a good idea to use puppet. I have been doing a lot research and so far I am convinced that Puppet would be very useful for us to have. However, I wanted to know, just like how you can run (in you example) NTP on all the servers on the domain; is it possible that you can run certain deamons on some blades and not on the others. In other words, if I don’t want a certain application to run on every blade I have how can I make that happen. Thank you again for all of your support.

7

Thanks a lot for this wonderful article. Our team here is looking to see whether if this is a good idea to use puppet. I have been doing a lot research and so far I am convinced that Puppet would be very useful for us to have. However, I wanted to know, just like how you can run (in you example) NTP on all the servers on the domain; is it possible that you can run certain deamons on some blades and not on the others. In other words, if I don’t want a certain application to run on every blade I have how can I make that happen. Thank you again for all of your support.

Error: Could not retrieve catalog from remote server: Error 400

I tried the example and getting :
puppet agent —test —server=`hostname`
Info: Retrieving plugin
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find class ntp for puppetmaster.ce.ad on node puppetmaster.ce.ad
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

Error experienced with the tutorial

I followed the tutorial and came across an error when trying the puppet agent —test —server=’hostname’.

The error is this:
err: Could not retrieve catalog from remote server: getAddrinfo: Name or service not known
warning: Not using cache failed catalog
err: could not retrieve catalog; skipping run

I retraced my steps and enter puppet master —mkusers —no-daemonize to see what is happening and got this message:
Could not run: Got 1 failure(s) while initializing: change from absent to directory failed: Could not set ‘directory on ensure: Permission denied - /var/lib/puppet/rrd

I am new to working with ubuntu and googled a possible solution and it said i had to change the /etc/hosts file by adding:

127.0.0.1 puppetd
127.0.0.1 puppet-master

But still got the error when trying to re execute the code

This tutorial's age

FYI to folks using this tutorial- with current versions of puppet (2013), it doesn’t appear to work as advertised. Something to do with the way the NTP service is being defined here I guess.

For a more current tutorial maintained by the developer, see here : http://docs.puppetlabs.com/learning/

I had much better luck there (I ignored the VM requirements and just ran on a CENTOS 6.3 box I had lying around).

Question about Puppet

I have deployed puppet in an environment where each client can be part of one two or three different environments. Is it possible to declare three diffrent environments in puppet client conf. If not is there any other way other than changing the config file and running the client to apply changes.

Formatting question with your example ntp.pp file

Here's your file from above:

class ntp {

package { "ntp":
ensure => installed
}

service { "ntp":
ensure => running,
}
}

Why is there a comma after ensure => running but not after ensure => installed?

puppet config for redhat linux

Hello

I have followed the manual but i got the below response when i tried to check the puppet agent..

[root@vm-sriappal-003 puppet]# puppet agent —test —server=`vm-sriappal-003`
-bash: vm-sriappal-003: command not found
err: Could not retrieve catalog from remote server: Connection refused - connect(2)
warning: Not using cache on failed catalog
err: Could not retrieve catalog; skipping run

i dont see the puppet service is running..
[root@vm-sriappal-003 puppet]# service puppetmaster status
puppetmaster: unrecognized service
[root@vm-sriappal-003 puppet]#

Please suggest me to make it work..

Thank you
 Srinivas

Post new comment

The content of this field is kept private and will not be shown publicly.
CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.
Image CAPTCHA
Enter the characters shown in the image.
By submitting this form, you accept the Mollom privacy policy.