Skip navigation.

Sysadmins Take Note

After many years as a professional sysadmin, my best tip for those just starting out is: take notes. Our job is basically about solving hard problems. (If they were easy, users would have solved them already.) This being so, keeping a detailed notebook of everything you do is like having an external brain pack. You don’t have to remember or rediscover how you solved that bizarre DNS problem back at your last gig; it’s in the book. You don’t have to reread the man page for half an hour to remind yourself how to self-sign an SSL certificate; it’s in the book. And you don’t have to waste time wondering where you got to on the current project at the end of last week; it’s in the book.

How to keep a notebook

The sysadmin notebook is one place technology won’t help you. A simple text file is best; you can check it into version control, and access it everywhere. It’s searchable, and if you organise your entries by date, you can find stuff that way as well.

I don’t recommend using a paper notebook, as one of the most useful functions of my notebook is to copy and paste information into it directly from the console session, or from web pages, and then to copy that information out again to save retyping. If you need your notebook with you when you’re on the move, try keeping the text file in your Dropbox and accessing it from your smartphone that way. Alternatively, if you keep it as a Simplenote document, you can sync it with Simplenote on your mobile device.

What to put in it

Working scientists keep a lab notebook in which they write down everything they did, and what the results were. Sysadmin work is also about careful, scientific, detailed experimentation, especially when troubleshooting. When there are many combinations of things to try, you need to document which ones you’ve already tried, and what happened. You can copy and paste console output directly into your notebook for later reference.

One handy tip is to record your console sessions with the script command, for later reference. For example:

$ script session.txt
Script started, output file is session.txt
bash-3.2$ echo Hello, world
Hello, world
bash-3.2$ exit
^F
Script done, output file is session.txt

$ cat session.txt
Script started on Mon Sep 27 17:30:40 2010
bash-3.2$ echo Hello, world
Hello, world
bash-3.2$ exit
^F
Script done on Mon Sep 27 17:30:52 2010

Once the job is completed, you can use the session file as a reference for writing up your notes, including copying and pasting commands and output where necessary.

A record of what you did

If you’re a freelancer, it’s important to be able to justify the time you’ve spent on a client project, and to be able to account for what you did and why. If you’re on salary, it’s even more important to be able to communicate regularly and accurately with the rest of the organisation, explaining what you do, and letting management know how valuable you are. The notebook is ideal for ‘weekly achievements’ meetings and emails, and a goldmine when it comes to annual review time.

A valuable reference

The nature of our work tends to involve complicated, lengthy processes requiring accurate command-line input and configuration settings, which take time to get right. This is where the notebook really scores: you need never rely on memory again. Having gone through the process of trial and error to get a system working, you can go back through your notes and distil them into a ‘how to do it’ document which will help to share knowledge with your co-workers, and form part of your essential procedure documentation.

A project management tool

You can also use it to store ‘things to do’ tasks and other reminders (keeping those at the bottom ensures that you see them every time you write a new entry). When you meet with clients or management to discuss project status, your notebook is an essential reference for what you’ve done, problems and roadblocks you’ve encountered, questions that need answering, and tasks that still remain to be done.

A communication tool

Wherever two or more sysadmins are gathered, confusion and problems can result from them all working on the same system - either at the same time, from remote locations, or consecutively, in a shift system. A common document where everybody details what they did and when, can be very useful. When something goes wrong it’s essential to be able to trace through all the changes that were made recently, and find out who or what broke the system.

Anyone who works in a team (and we all do, whether we realise it or not) needs a way to share information with their colleagues about what’s happening, and a team notebook is an excellent way to do this.

Examples

Here’s a couple of illustrative entries from my own notebook:

# 2009-10-09 Friday

Got Cribbage Corner site migrated from HostMonster and ran upgrade
script OK, but now having some trouble delegating the DNS from HM to
DynDNS.

Need to accept mail on peterman for help@cribbagecorner.com so trying
out sendmail virtusertable as detailed here:

    http://patriot.net/~scoile/isp-redhat/sendmail/virtusertable

This works fine so email to CC now works - everything is ready to
delegate the domain, but HostMonster doesn't accept the DynDNS
nameservers. Possibly it just hasn't been loaded yet on DynDNS so will
try again later.

Now fixed - the right sequence was:

    Set up the domain on DynDNS and 'pre-activate' it ('Force
    activation') 
    
    Wait for DynDNS to load the records:

	$ host cribbagecorner.com 204.13.249.76 
        Using domain server:
	Name: 204.13.249.76 
        Address: 204.13.249.76#53 
        Aliases:

	cribbagecorner.com has address 94.76.216.167
	cribbagecorner.com mail is handled by 10
	peterman.bitfieldconsulting.com.

    Then go to HostMonster and enter the nameservers:

	ns1.mydyndns.org ns2.mydyndns.org

    It didn't accept them as IP addresses which is why it kept saying
    'Invalid nameserver'. Dumb or what?

I found when the DNS was delegated over, accessing the main page as a
normal user gave 'Access denied' - though subpages worked OK. Found
this page:

    http://drupal.org/node/492734

which suggests going to Site Configuration / Performance and clicking
'Clear cached data'. This worked!

# 2010-09-08 Wednesday

Working on PHP build. Downloaded PHP-FPM:

  http://launchpad.net/php-fpm/master/0.6/+download/php-fpm-0.6~5.3.0.tar.gz
  
Generated patch:

  php-fpm-0.6-5.3.0/generate-fpm-patch
  
Downloaded PHP:

  http://us.php.net/get/php-5.3.0.tar.gz/from/us.php.net/mirror
  
Patched:

  patch -p1 < ../fpm.patch
  
Prepared for Debian build:

  dh_make -e sysadmin@example.com -f ../php-5.3.0.tar.gz
  
Edited control file and added custom configure line to debian/rules:

  override_dh_auto_configure:
          ./buildconf --force
          ./configure --with-fpm \
            --with-libevent=shared,/usr/local/lib/libevent.so \
            --with-mcrypt \
            --with-zlib \
            --enable-mbstring \
            --with-openssl \
            --with-mysql=/usr/bin/mysql_config \
            --with-mysqli=mysqlnd \
            --with-pdo-mysql=mysqlnd \
            --with-mysql-sock \
            --without-sqlite \
            --with-curl

per

  http://www.debian.org/doc/maint-guide/ch-dreq.en.html
  
Then:

  dpkg-buildpackage -b -uc -rfakeroot

In general, whenever I have to expend brainpower to solve a problem, I’ll consider it worth recording the solution (and the problem) in the notebook. Similarly, if there’s a command line that took some trial and error to construct, it’s worth a couple of seconds to copy and paste it into the notebook.

Start today

It doesn’t matter if you haven’t been keeping notes up to now and think it’s too late to start. It’s not. Just open a new text file, put the date, and type in what you did today.

Feedback

Do you keep a notebook? Do you use a paper book, a text file, or special software? A web service? What do you use your notes for? Have they ever saved your bacon? Do you use a wiki or other shared document tool to communicate with your team or co-workers? Do you have any other insider tips for sysadmins? Let us know in the comments.

org-mode

Emacs + Org-mode is great for this :)

Re: org-mode

Cool! Does it have any special features which help with the note taking?

‘C-c C-r’ pops up a new

C-c C-r’ pops up a new buffer to take a note, ‘C-c C-c’ to confirm / ‘C-c C-k’ to cancel. They’re fed into your default notes file (see below).

I have IRC and Music (emms) running in emacs as well, so unless I’m in a browser, I’m always (basically) in emacs anyway.

It has neat dates tamps with a calendar picker on ‘(C-u) C-c .’, which I mostly use for journaling. I try and do pomodoro-style 25 minute journal markers.

This is my org config: http://gist.github.com/599521

I used to be a real

I used to be a real Emacs-o-phile too! I used it for coding, email, web browsing, newsreading, and making tea!

I discovered TextMate on the Mac though so I have gone over to the dark side.

Been using a Tiddlywiki for

Been using a Tiddlywiki for that for a couple of years now, although I’m rarely that detailed unless it a real pain of an issue or one I expect to reoccur. It’s my todo list as well. Handy!

I forgot about Tiddlywiki! I

I forgot about Tiddlywiki! I used to use that for a while!

And have you any success stories to tell from keeping your notebook?

Evernote

Love evernote for capturing all my notes. Web/Mac/Windows clients.

MacJournal

I used text files for years but settled with MacJournal. Really great product for keeping all kinds of notes and personal documentation. You don’t end up with hundreds of files, it has a built-in fulltext search, tags, labels (colors) and you can keep all/selected journals in sync with your iPhone or iPad.

Use a blog! This way your

Use a blog!

This way your little “this worked for me” notes-to-self show up in google for the rest of us to crib off of. You’ll even occasionally get a commenter offering a tip or improvement.

I use ELOG

What I have found works well for me is an application called ELOG

https://midas.psi.ch/elog/#screenshots

Its advantage is that it provides excellent search features across its log books
so you can easily find that esoteric solution that you documented many moons ago.

Or if you use it to log incidents or changes on live infrastructure its useful
for targeted searches to understand why things keep happening or why something bad
has recently happened. When I have worked on client projects I have configured the
email feature so that the client can receive emails when I log important milestones
on the project.

Google Notebook is no longer in active development

Unfortunately, Google Notebook is no longer open to new sign-ups; you can only use it if you created an account prior to development being ceased.

the Personal LOgging Device

I use plod - more specifically the modified Rouillard version (http://www.cs.umb.edu/~rouilj/#plod) and not the original by Hal (http://www.deer-run.com/~hal/).

Great for logging actions and making notes, and the modified version also keeps track of time worked on your projects!

Simplenote

I’ve been using the Dropbox/Simplenote method for quite a while now. Syncs across all machines & iPhones, etc but I’ve usually used Vim for it. Hadn’t come across script before so I’ll give it a try…

Yes that’s right

Yes that’s right communication tool is really important in this scenario. We usually have to working as a team and there should be a document available in every access.

Post new comment

The content of this field is kept private and will not be shown publicly.
CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.
Image CAPTCHA
Enter the characters shown in the image.
By submitting this form, you accept the Mollom privacy policy.