For me, what is really exciting about DevOps is the notion that software development, infrastructure engineering and operational automation can and should done simultaneously and collaboratively. DevOps doesn’t invalidate ITIL, nor does it mean unbridled application deployment.

The way I see it we already have some great examples of DevOps in action — its just that the term hasn’t been applied to them yet.

Can you remember the last time when you had to apply patches or config file changes to a system. And did you have that fingers crossed feeling? Wouldn’t it be great that you could install a patch and run a series of tests to see if everything behaved the way it should?

I’ve been managing systems teams in an Agile environment for a number of years, and after thought and experimentation, I can recommend using an approach borrowed from Lean systems management, called Kanban.

Here is a detailed example of a fairly typical 2-tier Kanban board, for teams that know the basics of Kanban and are taking their first steps towards implementing it in practice.

Agile training expert Robert Dempsey examines the application of Kanban techniques to system administration work, with a devops flavour.

Cassandra is a highly scalable NoSQL database solution. Jonathan Ellis demolishes some misconceptions about Cassandra with a look at its replication capabilities, reliability, and interoperability with Hadoop and similar tools.

A review of the various HA solutions available for MySQL.

Gizzard is a Scala framework that makes it easy to create custom fault-tolerant, distributed databases. At a high level, Gizzard is a middleware networking service that manages partitioning data across arbitrary backend datastores (e.g., SQL databases, Lucene, etc.). Nick Kallen from Twitter Engineering outlines what Gizzard is and how it works.

Handy notes and tips on squeezing more performance out of your MySQL server.

Hive is a data warehouse infrastructure built over Hadoop. It provides tools to enable easy data ETL, a mechanism to put structures on the data, and the capability to querying and analysis of large data sets stored in Hadoop files.

David Mytton of BoxedIce explains how his MongoDB infrastructure works and gives some tips and tricks based on experience running MongoDB in production for several months.

SQL databases are fundamentally non-scalable, and there is no magical pixie dust that we, or anyone, can sprinkle on them to suddenly make them scale.

In this FutureRuby talk, Ilya Grigorik explores Tokyo Cabinet’s features such as the key-value store, ordered traversal, attribute search, schemaless data structures,indexing, and scripting with Lua.

The two most important part of this talk is the observation that Dev, Qa and Operations teams have to slightly blend into each other to achieve deployments at such a velocity, and the fact that they are not afraid to break the website by deploying code from trunk.

More from Agile Testing’s Grig Gheorghiu on setting up Opscode Chef, creating your own cookbook, modifying an existing cookbook, creating a role and adding a client machine to that role.

How to set up continuous integration testing for Puppet manifests.

It’s been said that deploying Java apps is hard for Linux packages, but Puppet makes it very easy. This is only the tip of the iceberg—you can use the same tool to deploy mailservers and databases as well as appservers. It fits in well whether you have 20 machines or thousands. It’s agnostic to cloud vs physical hardware, and plays nicely in all places. The example that follows below is designed to be executed locally, though in a typical deployment, you’ll host it on a central server, called a puppetmaster, and then roll your configuration out to the nodes using puppet.

It’s important to realize that the tools you use are largely independent of the integration strategy you use. Although many people associate DVCSs with feature branching, they can be used with CI. All you need to do is mark one branch on one repository as the mainline. If everyone pulls and pushes to that every day, then you have a CI mainline. Indeed with a disciplined team, I would usually prefer to use a DVCS on a CI project than a centralized one.

Agile Testing’s Grig Gheorghiu provides a simple, step-by-step tutorial for installing Opscode Chef and getting a Chef client to talk to a Chef server.

Julian Simpson (aka @builddoctor) is a well-known authority on software build and deployment issues, including configuration management, continuous integration, continuous deployment, and testing. His blog covers all of these topics, featuring talks, interviews, links, and patterns and practices.

Vagrant is a tool for building and distributing virtualized development environments, providing easy to configure, lightweight, reproducible, and portable virtual machines targeted at development environments. Vagrant includes automated virtual machine creation using Oracle’s VirtualBox, automated provisioning of virtual environments using Opscode Chef, port forwarding to the host machine, full SSH access to created environments, shared folders, packaging environments into distributable boxes, and easy teardown and rebuild of your environments.

For me, what is really exciting about DevOps is the notion that software development, infrastructure engineering and operational automation can and should done simultaneously and collaboratively. DevOps doesn’t invalidate ITIL, nor does it mean unbridled application deployment.

The way I see it we already have some great examples of DevOps in action — its just that the term hasn’t been applied to them yet.

How to set up continuous integration testing for Puppet manifests.

Deployment is definitely one of the places where the rubber meets the road. In some organizations, deployment of new code can be the single-most stressful and dividing parts of their work.

Devopsdays was a small conference about a couple of emerging themes combining Development and Operations.

Hilarious Monty-Python-mashup video explaining the devops movement.

Video of Stephen Nelson-Smith speaking about Git for sysadmins - how it works, how to use it, how to support it, and how it compares to other version control systems.

Julian Simpson (aka @builddoctor) is a well-known authority on software build and deployment issues, including configuration management, continuous integration, continuous deployment, and testing. His blog covers all of these topics, featuring talks, interviews, links, and patterns and practices.

Ten characteristics of devops-flavoured sysadmin practices, as outlined by Dmitriy Samovskiy, including coding skills, protection of business revenues, a focus on stability and uptime, and distributed or hyper-distributed applications.

Agile training expert Robert Dempsey examines the application of Kanban techniques to system administration work, with a devops flavour.

If your dev team is fixing servers, they are not programming. If they are not programming, they may not be making you money. Many developers needlessly use versions of software that are not part of the standard OS. Web developers often lack the experience to know the accepted standards. As a result, you may end up with “odd” deployment scenarios which can complicate management.

More from Agile Testing’s Grig Gheorghiu on setting up Opscode Chef, creating your own cookbook, modifying an existing cookbook, creating a role and adding a client machine to that role.

Excellent introduction to Puppet for cloud management by Jeff Wallace, featuring EC2 and Rackspace Cloud integration, Puppet classes, using Ruby logic in ERB templates, and inheriting node definitions to create identical configurations.

Using these tools should help you build more reliable and less noisy cron jobs, which makes your systems more reliable and your pager more quiet.

A “design pattern” is a little bit of knowledge that is worth sharing, and useful for repeating. For example, the design pattern of “a network” or “active-active redundancy” or “RAID-0″ or “helpdesk”. Ever visit an IT shop where there was no way for people to get help? I have. So I explained the design pattern of a “helpdesk” and without having to re-invent the wheel, they were able to implement one.

When planning the storage for a system one thing you need to know above all other information is how that data is going to be accessed. I’m talking about 95% reads 5% writes, 1.2Gb/minute average transfer, highly latency sensitive. You can make some assumptions based on the applications that’ll be accessing the storage, but if you really need to know, the only way to find out is measuring.

Once you know how that data is going to be accessed, you can build or provision its storage accordingly. Knowing how likely the dataset is to grow is also something you need to know, but that’s a luxury we often don’t get. And for the love of performance metrics, don’t forget peak loading and behavior under fault conditions.

Twitter recently switched from an Apache/Mongrel web infrastructure to one based on Unicorn, a high-performance load balancing application server, and Stormcloud, a Unicorn monitoring and management system. Ben Sandofsky and John Adams from Twitter Engineering explain the benefits.

Julian Simpson (aka @builddoctor) is a well-known authority on software build and deployment issues, including configuration management, continuous integration, continuous deployment, and testing. His blog covers all of these topics, featuring talks, interviews, links, and patterns and practices.

Poul-Henning Kamp is the author of Varnish, an open-source HTTP accelerator used by Facebook, Slashdot, Wikia and many other sites to speed up slow Web servers. In this article for ACM Queue he describes how he applied his experience as a lead kernel developer on FreeBSD to developing a new, ultra-high-performance algorithm for cache expiry, and discusses how to optimise traditional algorithms for virtual-memory execution environments.

“Wouldn’t it be nice if we could abstract some of the low-level details of different socket types, connection handling, framing, or even routing? This is exactly where the ZeroMQ (ØMQ/ZMQ) networking library comes in: “it gives you sockets that carry whole messages across various transports like inproc, IPC, TCP, and multicast; you can connect sockets N-to-N with patterns like fanout, pubsub, task distribution, and request-reply”.” Ilya Grigorik explains the architecture of ZeroMQ with some code examples and diagrams.

Version control and other development best practices which sysadmins should use.

For years, IT pros have heard that they must do more with less, as staffing is cut and outsourced, while demands to better serve the business and adopt new technologies continually increase. This is how it’s always been, but it doesn’t have to be how it always will be.

Let’s face it, System Administrators get no respect 364 days a year. This is the day that all fellow System Administrators across the globe, will be showered with expensive sports cars and large piles of cash in appreciation of their diligent work.

Build, deployment and infrastructure superdude Julian Simpson has a job site specifically for software build and deployment professionals. Very useful if you’re looking for a position in the industry - or if you want to advertise a vacancy.

Agile training expert Robert Dempsey examines the application of Kanban techniques to system administration work, with a devops flavour.

Many of my former Go students headed to Berkeley and Stanford, chose computer science major, and became software engineer. At the minimal, Go will calm a person down and teaches him/her to concentrate for a long period of time.

Lock-picking will teach you to be humble and not focus on what others can do better, but to take an honest inventory of your skills and your progress.

Here are a few things you need to tweak in order to improve OpenSSH server security.

cucumber-puppet is the glue between cucumber and Puppet, allowing you to write behavioural tests, or features as cucumber calls it, for your Puppet manifest.

DNS Knife is a good tool to check if your DNS setup is ok, it checks the parent servers, it checks for if your nameservers are listed on the parent server, checks if all your nameservers are reachable and are authoritative. And so on and so on…

Video of Stephen Nelson-Smith speaking about Git for sysadmins - how it works, how to use it, how to support it, and how it compares to other version control systems.

A simple command-line tool to be able to grab real-time stats from memcache.

pigz, which stands for parallel implementation of gzip, is a fully functional replacement for gzip that exploits multiple processors and multiple cores to the hilt when compressing data. pigz was written by Mark Adler, and uses the zlib and pthread libraries. John Allspaw notes that “When you’ve got massive files, this can be a pretty big advantage, especially when you’ve got lots of cores sitting around.”.

Agile Testing’s Grig Gheorghiu provides a simple, step-by-step tutorial for installing Opscode Chef and getting a Chef client to talk to a Chef server.

A large crowdsourced list of sysadmin diagnostic tools available on Linux.

Masterzen explains how to offload the job of Puppet static file serving onto Nginx, a small and performant web server, and also how to have Nginx cache the files. He also gives a recipe for configuring Nginx to cache the compiled catalogs for each machine, reducing some of the compute load on the Puppet server.

think it makes sense to use a tool like Puppet for the initial configuration of the OS and of the packages required by your application. When it comes time to deploy your application, I think a tool like Fabric is more appropriate.

So far, my biggest problem with Nagios has been finding the time to add new systems to it, figuring out what services to check, etc. It’s not a particularly difficult thing to do, but in the grand scheme of things, it was just something that always fell by the wayside in the drive to get more systems set up, deal with user problems, and put out the inevitable fires. That is, until recently.

How to set up continuous integration testing for Puppet manifests.

Excellent introduction to Puppet for cloud management by Jeff Wallace, featuring EC2 and Rackspace Cloud integration, Puppet classes, using Ruby logic in ERB templates, and inheriting node definitions to create identical configurations.

cucumber-puppet is the glue between cucumber and Puppet, allowing you to write behavioural tests, or features as cucumber calls it, for your Puppet manifest.

It’s been said that deploying Java apps is hard for Linux packages, but Puppet makes it very easy. This is only the tip of the iceberg—you can use the same tool to deploy mailservers and databases as well as appservers. It fits in well whether you have 20 machines or thousands. It’s agnostic to cloud vs physical hardware, and plays nicely in all places. The example that follows below is designed to be executed locally, though in a typical deployment, you’ll host it on a central server, called a puppetmaster, and then roll your configuration out to the nodes using puppet.

People complain that Puppet is non-deterministic. On a certain level that is like complaining that threads are non-deterministic. That’s the way the model works by design. If there is logic that depends on the order of execution, that code needs mutex/syncronization. Threads create issues, but they also solve some.

Great article on configuring the Puppet server to use Passenger and Apache, with a complete example vhost definition for puppetmasterd.

Setting up a staging environment for your modules, manifests and files on the puppetmaster. Test your modules, manifests, templates files and facts before deploying on production servers.

More from Agile Testing’s Grig Gheorghiu on setting up Opscode Chef, creating your own cookbook, modifying an existing cookbook, creating a role and adding a client machine to that role.

Kokki is a configuration management framework inspired by Chef.

Moonshine is an opensource configuration management and deployment system that follows the Rails way, simplifying server configuration, dependency management, and Rails application deployment, using Ruby and Puppet.

Agile Testing’s Grig Gheorghiu provides a simple, step-by-step tutorial for installing Opscode Chef and getting a Chef client to talk to a Chef server.

slaughter is a simple tool which will allow policies to be downloaded, via HTTP, from a central server and executed upon a local machine. The intention is that these local policies may be written in a portable fashion and used to automate the administration of a large number of Linux machines.

Sprinkle is based on capistrano and uses the same push model without any additional infrastructure.

Vagrant is a tool for building and distributing virtualized development environments, providing easy to configure, lightweight, reproducible, and portable virtual machines targeted at development environments. Vagrant includes automated virtual machine creation using Oracle’s VirtualBox, automated provisioning of virtual environments using Opscode Chef, port forwarding to the host machine, full SSH access to created environments, shared folders, packaging environments into distributable boxes, and easy teardown and rebuild of your environments.

Movist is a fast and user-friendly media player for OS X, based on FFMPEG and Quicktime.

SynergyKM is a GUI wrapper around the synergy command line tool that lets you easily share a single mouse and keyboard between multiple computers with different operating systems without special hardware.

Drucumber is a new module that converts native-language like text into a Drupal Simpletests. The goal of this module is to allow end-users and project teams to implement behavior driven development: specify behaviors or features of a Drupal site in a test and never again worry about checking it.

n the two years since I’ve used Mollom, the service probably has blocked more than 100,000 pieces of spam from being posted at my site.

Basically it adds a hidden form element which your users won’t see, but automated spambots will. If anything is placed in the textfield, validation fails. So instead of making the user prove that they are human, we allow the spambot to betray itself as a bot.

Taking advantage of Nginx’s direct Memcache integration by delivering Drupal pages from cache - even for authenticated users.

There are many ways to improve the performance & scalability of Drupal. Comparison of a selection of performance and scalability modules.

Lifetime Digital case study highlights the proven scalability and performance of Drupal websites in a high traffic environment. By utilizing Akamai technology for caching of rich media and for anonymous users, Lifetime Digital’s 6 Drupal sites have proven successful in supporting 50 million page views per month.

As anyone who has developed a Drupal site with devel module’s query logging on can tell you, a Drupal implementation can quickly get out of control when it comes to hammering the database.