Scripting with Go

From The Power of Go: Tools

The Unix shell is pure wizardry. With the right incantation of shell spells, you can organise files, process text, compute data, and feed the output of any program to the input of any other. We might even say, paraphrasing Clarke’s Third Law:

Any sufficiently clever shell one-liner is indistinguishable from magic.

In this article we’ll explore a package called script that aims to sprinkle some Unix shell fairy dust on your Go programs.

Why Go?

If the shell is the traditional way of writing systems software, then what would be the point of using a language like Go instead? It has many advantages: Go programs are fast, scalable, can be written quickly, and can also be maintained by large teams over a long time.

For example, consider a typical devops task such as counting the lines in a log file that match a certain string (error, let’s say). Most experienced Unix users would write some kind of shell one-liner to do this:

grep error log.txt |wc -l

A typical task

But shell wizards (and witches) can do much more. For example, suppose we have a web server access log to analyse. Here’s a typical line from such a log: it contains the client’s IP address, a timestamp, and various information about the request.

203.0.113.17 - - [30/Jun/2019:17:06:15 +0000] "GET / HTTP/1.1" 200 2028 "https://example.com/ "Mozilla/5.0..."

And suppose we want to find the top ten most frequent visitors to our website, by IP address. How could we do that?

Each line represents one request, so we’ll need to count the lines for each IP address, sort them in descending order of frequency, and take the first ten.

A shell one-liner like this would do the job:

cut -d' ' -f 1 access.log |sort |uniq -c |sort -rn |head

This extracts the IP address from the first column of each line (cut), counts the number of unique values (uniq -c), sorts them in descending numerical order (sort -rn), and shows the first ten (head).

So can we do the same sort of magic in Go? Let’s try to write a similar program and see how easy it is (or not).

Programs as pipelines

Given the nature of the problem, we’d like to express the solution as a data pipeline, just like the shell program. How could we express that in Go? What about something like this?

script.File("log.txt").Column(1).Freq().First(10).Stdout()

In other words, read the file log.txt, take its first column, sort by frequency, get the first ten results, and print them to standard output.

The script package

This example uses a Go package called script:

import "github.com/bitfield/script"

Let’s see a few more shell-like tasks we can do succinctly in Go using script. Suppose you want to read the contents of a file as a string, for example:

data, err := script.File("test.txt").String()

Or count the number of lines it contains:

n, err := script.File("test.txt").CountLines()

How about counting only lines that contain the string “Error”?

n, err := script.File("test.txt").Match("Error").CountLines()

Similarly, we can read lines from the program’s standard input and filter by string matching:

script.Stdin().Match("Error").Stdout()

Or read from a list of files supplied as the program’s command-line arguments:

script.Args().Concat().Match("Error").Stdout()

We’re not limited to getting data only from files or standard input. We can get it from HTTP requests too:

script.Get("https://wttr.in/London?format=3").Stdout()
// Output:
// London: 🌦   +13°C

If the response is in JSON format, we can use JQ queries to interrogate it:

data, err := script.Do(req).JQ(".[0] | {message: .commit.message, name: .commit.committer.name}").String()

We can also run external programs and capture their output:

script.Exec("ping 127.0.0.1").Stdout()

Note that Exec runs the command concurrently: it doesn’t wait for the command to complete before returning any output. That’s good, because this ping command will run forever (or until we get bored).

Userland tools

One of the things that makes shell scripts powerful is not just the shell language itself, which is pretty basic. It’s the availability of a rich set of userland tools, like grep, awk, cat, find, head, and so on.

But we can replicate most of the functionality of those tools using script, which means that we can run our Go binary in an environment where no userland tools are installed, such as a scratch container.

For example, here’s a program that just echoes its input to its output, like cat:

script.Stdin().Stdout()

And here’s one that concatenates all the files it’s given as arguments and writes them to the output, again like cat:

script.Args().Concat().Stdout()

One common operation in shell scripts is to use the find tool to generate a recursive directory listing. We can do that too:

script.FindFiles("/backup").Stdout()

But supposing we then wanted to do some operation on each of the files discovered in this way. What would that look like?

script.FindFiles("*.go").ExecForEach("gofmt -w {{ . }}")

You might recognise the argument to ExecForEach as a Go template; every filename produced by FindFiles will be substituted into this command in turn.

Everything is a pipe

All script programs are pipelines, as we’ve seen. These usually begin with some source of data, such as a file:

p := script.File("test.txt")

You might expect File to return an error if there is a problem opening the file, but it doesn’t. We will want to call a chain of methods on the result of File, and it’s inconvenient to do that if it also returns an error. So File returns a pipe instead.

Since File returns a pipe, you can call any method on it you like. For example, Match:

p.Match("what I'm looking for")

The result of this is another pipe (containing only the matching lines from test.txt), and so on. You don’t have to chain all your methods onto a single line, but it’s pretty neat that you can if you want to.

Handling errors

The pipeline idea means we don’t have to check errors after each individual stage. Instead, we can check for an error at the end of the pipe, by calling its Error method:

if err := p.Error(); err != nil {
    return fmt.Errorf("oh no: %w", err)
}

This eliminates a lot of the if err != nil boilerplate which seems to infuriate people so much about Go. Those people aren’t crazy: that kind of thing can be annoying, if you don’t know this useful pattern.

Scripting with Go

The script package is implemented entirely in Go, and does not require any external userland programs. Thus you can build your script program as a single (very small) binary that is quick to build, quick to upload, quick to deploy, quick to run, and economical with resources.