Sysadmins Take Note
After many years as a professional sysadmin, my best tip for those just starting out is: take notes. Our job is basically about solving hard problems. (If they were easy, users would have solved them already.) This being so, keeping a detailed notebook of everything you do is like having an external brain pack. You don’t have to remember or rediscover how you solved that bizarre DNS problem back at your last gig; it’s in the book. You don’t have to reread the man page for half an hour to remind yourself how to self-sign an SSL certificate; it’s in the book. And you don’t have to waste time wondering where you got to on the current project at the end of last week; it’s in the book.
How to keep a notebook
The sysadmin notebook is one place technology won’t help you. A simple text file is best; you can check it into version control, and access it everywhere. It’s searchable, and if you organise your entries by date, you can find stuff that way as well.
I don’t recommend using a paper notebook, as one of the most useful functions of my notebook is to copy and paste information into it directly from the console session, or from web pages, and then to copy that information out again to save retyping. If you need your notebook with you when you’re on the move, try keeping the text file in your Dropbox and accessing it from your smartphone that way. Alternatively, if you keep it as a Simplenote document, you can sync it with Simplenote on your mobile device.
What to put in it
Working scientists keep a lab notebook in which they write down everything they did, and what the results were. Sysadmin work is also about careful, scientific, detailed experimentation, especially when troubleshooting. When there are many combinations of things to try, you need to document which ones you’ve already tried, and what happened. You can copy and paste console output directly into your notebook for later reference.
One handy tip is to record your console sessions with the script command, for later reference. For example:
$ script session.txt Script started, output file is session.txt bash-3.2$ echo Hello, world Hello, world bash-3.2$ exit ^F Script done, output file is session.txt $ cat session.txt Script started on Mon Sep 27 17:30:40 2010 bash-3.2$ echo Hello, world Hello, world bash-3.2$ exit ^F Script done on Mon Sep 27 17:30:52 2010
Once the job is completed, you can use the session file as a reference for writing up your notes, including copying and pasting commands and output where necessary.
A record of what you did
If you’re a freelancer, it’s important to be able to justify the time you’ve spent on a client project, and to be able to account for what you did and why. If you’re on salary, it’s even more important to be able to communicate regularly and accurately with the rest of the organisation, explaining what you do, and letting management know how valuable you are. The notebook is ideal for ‘weekly achievements’ meetings and emails, and a goldmine when it comes to annual review time.
A valuable reference
The nature of our work tends to involve complicated, lengthy processes requiring accurate command-line input and configuration settings, which take time to get right. This is where the notebook really scores: you need never rely on memory again. Having gone through the process of trial and error to get a system working, you can go back through your notes and distil them into a ‘how to do it’ document which will help to share knowledge with your co-workers, and form part of your essential procedure documentation.
A project management tool
You can also use it to store ‘things to do’ tasks and other reminders (keeping those at the bottom ensures that you see them every time you write a new entry). When you meet with clients or management to discuss project status, your notebook is an essential reference for what you’ve done, problems and roadblocks you’ve encountered, questions that need answering, and tasks that still remain to be done.
A communication tool
Wherever two or more sysadmins are gathered, confusion and problems can result from them all working on the same system - either at the same time, from remote locations, or consecutively, in a shift system. A common document where everybody details what they did and when, can be very useful. When something goes wrong it’s essential to be able to trace through all the changes that were made recently, and find out who or what broke the system.
Anyone who works in a team (and we all do, whether we realise it or not) needs a way to share information with their colleagues about what’s happening, and a team notebook is an excellent way to do this.
Here’s a couple of illustrative entries from my own notebook:
# 2009-10-09 Friday Got Cribbage Corner site migrated from HostMonster and ran upgrade script OK, but now having some trouble delegating the DNS from HM to DynDNS. Need to accept mail on peterman for firstname.lastname@example.org so trying out sendmail virtusertable as detailed here: http://patriot.net/~scoile/isp-redhat/sendmail/virtusertable This works fine so email to CC now works - everything is ready to delegate the domain, but HostMonster doesn't accept the DynDNS nameservers. Possibly it just hasn't been loaded yet on DynDNS so will try again later. Now fixed - the right sequence was: Set up the domain on DynDNS and 'pre-activate' it ('Force activation') Wait for DynDNS to load the records: $ host cribbagecorner.com 22.214.171.124 Using domain server: Name: 126.96.36.199 Address: 188.8.131.52#53 Aliases: cribbagecorner.com has address 184.108.40.206 cribbagecorner.com mail is handled by 10 peterman.bitfieldconsulting.com. Then go to HostMonster and enter the nameservers: ns1.mydyndns.org ns2.mydyndns.org It didn't accept them as IP addresses which is why it kept saying 'Invalid nameserver'. Dumb or what? I found when the DNS was delegated over, accessing the main page as a normal user gave 'Access denied' - though subpages worked OK. Found this page: http://drupal.org/node/492734 which suggests going to Site Configuration / Performance and clicking 'Clear cached data'. This worked! # 2010-09-08 Wednesday Working on PHP build. Downloaded PHP-FPM: http://launchpad.net/php-fpm/master/0.6/+download/php-fpm-0.6~5.3.0.tar.gz Generated patch: php-fpm-0.6-5.3.0/generate-fpm-patch Downloaded PHP: http://us.php.net/get/php-5.3.0.tar.gz/from/us.php.net/mirror Patched: patch -p1 < ../fpm.patch Prepared for Debian build: dh_make -e email@example.com -f ../php-5.3.0.tar.gz Edited control file and added custom configure line to debian/rules: override_dh_auto_configure: ./buildconf --force ./configure --with-fpm \ --with-libevent=shared,/usr/local/lib/libevent.so \ --with-mcrypt \ --with-zlib \ --enable-mbstring \ --with-openssl \ --with-mysql=/usr/bin/mysql_config \ --with-mysqli=mysqlnd \ --with-pdo-mysql=mysqlnd \ --with-mysql-sock \ --without-sqlite \ --with-curl per http://www.debian.org/doc/maint-guide/ch-dreq.en.html Then: dpkg-buildpackage -b -uc -rfakeroot
In general, whenever I have to expend brainpower to solve a problem, I’ll consider it worth recording the solution (and the problem) in the notebook. Similarly, if there’s a command line that took some trial and error to construct, it’s worth a couple of seconds to copy and paste it into the notebook.
It doesn’t matter if you haven’t been keeping notes up to now and think it’s too late to start. It’s not. Just open a new text file, put the date, and type in what you did today.
Do you keep a notebook? Do you use a paper book, a text file, or special software? A web service? What do you use your notes for? Have they ever saved your bacon? Do you use a wiki or other shared document tool to communicate with your team or co-workers? Do you have any other insider tips for sysadmins? Let us know in the comments.