Masterzen explains how to offload the job of Puppet static file serving onto Nginx, a small and performant web server, and also how to have Nginx cache the files. He also gives a recipe for configuring Nginx to cache the compiled catalogs for each machine, reducing some of the compute load on the Puppet server.

Autobench is a simple Perl script for automating the process of benchmarking a web server (or for conducting a comparative test of two different web servers).

There are many ways to scale modern web applications. What I will be describing here is the method that we chose. This should by no means be considered the only way to scale an application. Consider it a case study of what worked for us given our unique requirements.

When planning the storage for a system one thing you need to know above all other information is how that data is going to be accessed. I’m talking about 95% reads 5% writes, 1.2Gb/minute average transfer, highly latency sensitive. You can make some assumptions based on the applications that’ll be accessing the storage, but if you really need to know, the only way to find out is measuring.

Once you know how that data is going to be accessed, you can build or provision its storage accordingly. Knowing how likely the dataset is to grow is also something you need to know, but that’s a luxury we often don’t get. And for the love of performance metrics, don’t forget peak loading and behavior under fault conditions.

EXT3, EXT4, XFS, ReiserFS, and Btrfs benchmarked.

How to use vmstat, iostat, and top to understand what part of your system is the bottleneck.

pigz, which stands for parallel implementation of gzip, is a fully functional replacement for gzip that exploits multiple processors and multiple cores to the hilt when compressing data. pigz was written by Mark Adler, and uses the zlib and pthread libraries. John Allspaw notes that “When you’ve got massive files, this can be a pretty big advantage, especially when you’ve got lots of cores sitting around.”.

Insight into the process of building scalable internet based services.

Twitter recently switched from an Apache/Mongrel web infrastructure to one based on Unicorn, a high-performance load balancing application server, and Stormcloud, a Unicorn monitoring and management system. Ben Sandofsky and John Adams from Twitter Engineering explain the benefits.

iperf is a great little tool for quickly measuring the network bandwidth between two machines.

Poul-Henning Kamp is the author of Varnish, an open-source HTTP accelerator used by Facebook, Slashdot, Wikia and many other sites to speed up slow Web servers. In this article for ACM Queue he describes how he applied his experience as a lead kernel developer on FreeBSD to developing a new, ultra-high-performance algorithm for cache expiry, and discusses how to optimise traditional algorithms for virtual-memory execution environments.

We have been doing various experiments in our ec2 web serving cluster to serve maximum traffic at the minimum costs. I thought our experience will be useful to many other people using ec2.