Testing CLI tools in Go

From The Power of Go: Tests

All your tests pass, but the program crashes when you run it. Sound familiar?

In Part 1 of this series we made friends with the very useful testscript package, and saw how to write and run test scripts from our Go tests.

If all we could do with testscript were to run existing programs with certain arguments, and assert that they succeed (or fail), and produce certain outputs (or not), that would still be pretty useful.

But we’re not limited to running existing programs. If we want to test our own binary, for example, we don’t have to go through all the labour of compiling and installing it first, in order to execute it in a script. testscript can save us the trouble. Let’s see how.

Suppose we’re writing a program named hello, for example, whose job is simply to print a “hello world” message on the terminal. As it’s an executable binary, it will need a main function. Something like this would do:

func main() {
    fmt.Println("hello world")
}

How can we test this? The main function can’t be called directly from a Go test; it’s invoked automatically when we run the compiled binary. We can’t test it directly.

So we’ll instead delegate its duties to some other function that we can call from a test: hello.Main, let’s say.

We’ll do nothing in the real main function except call hello.Main to do the printing, and then exit with whatever status value it returns:

func main() {
    os.Exit(hello.Main())
}

(Listing hello/1)

Fine. So we can deduce from this what the signature of hello.Main needs to be. It takes no arguments, and it returns an int value representing the program’s exit status. Let’s write it:

package hello

import "fmt"

func Main() int {
    fmt.Println("hello world")
    return 0
}

(Listing hello/1)

This is a very simple program, of course, so it doesn’t have any reason to report a non-zero exit status, but it could if it needed to. Here, we just explicitly return 0, so the program will always succeed.

Now that we’ve delegated all the real functionality of the program to this hello.Main function, we could call it from a Go test if we wanted to. But that wouldn’t invoke it as a binary, only as a regular Go function call, which isn’t really what we want here. For example, we probably want different values of os.Args for each invocation of the program.

What we need to do instead is to tell testscript to make our program available to scripts as a binary named hello. Here’s what that looks like in our Go test code:

func TestMain(m *testing.M) {
    os.Exit(testscript.RunMain(m, map[string]func() int{
        "hello": hello.Main,
    }))
}

(Listing hello/1)

The function TestMain is special to Go: it doesn’t test anything itself, but its job is usually to set something up in advance, before any test actually runs.

What this TestMain is doing is calling testscript.RunMain. What does that do? Well, it runs all our Go tests, but before it does that, it also sets up any custom programs that we want to use in scripts.

To do that, we pass this map to RunMain, connecting the name of our desired binary (hello) with its delegate main function (hello.Main):

map[string]func() int{
    "hello": hello.Main,
}

This tells testscript to create an executable binary named hello, whose main function will call hello.Main. This binary will be installed in a temporary directory (not the script’s work directory), and that directory will be added to the $PATH variable in the environment of all our scripts.

If the magic works the way it should, then, we’ll be able to use exec in a script to run the hello program, just as if we’d compiled and installed it manually. After the tests have finished, the binary and its temporary directory will be deleted automatically.

Let’s give it a try. We’ll create a script with the following contents:

exec hello
stdout 'hello world\n'

(Listing hello/1)

Now we’ll add a test that runs this script using testscript.Run, as before:

func TestHello(t *testing.T) {
    testscript.Run(t, testscript.Params{
        Dir: "testdata/script",
    })
}

(Listing hello/1)

Here’s the result of running go test:

PASS

It worked! But are we really executing the binary implemented by hello.Main, or does there just happen to be some unrelated program named hello somewhere on our system? You can’t be too careful these days.

To find out, let’s change the hello.Main function to print something slightly different, and see if that makes the test fail. This ought to prove that testscript is really running the program we think it is:

func HelloMain() int {
    fmt.Println("goodbye world")
    return 0
}

Here’s the result:

> exec hello
[stdout]
goodbye world
> stdout 'hello world\n'
FAIL: testdata/script/hello.txtar:2: no match for `hello world\n`
found in stdout

Proof positive that we’re executing the right hello program, I think you’ll agree. Let’s also check that returning anything other than 0 from hello.Main causes the exec assertion to fail, as we would expect:

func HelloMain() int {
    fmt.Println("hello world")
    return 1
}

Here’s the result:

> exec hello
[stdout]
hello world
[exit status 1]
FAIL: testdata/script/hello.txt:1: unexpected command failure

One thing to be careful of when defining custom commands in this way is to remember to call os.Exit with the result of testscript.RunMain. For example, suppose we were to write a TestMain like this:

func TestMain(m *testing.M) {
    testscript.RunMain(m, map[string]func() int{
        "hello": hello.Main,
    })
    // oops, forgot to use 'status'
}

This looks reasonable, but the status value returned by RunMain (which is the exit status of our custom command) is ignored. Implicitly, we exit with a zero exit status, meaning that the hello command would always appear to “succeed”, regardless of what hello.Main actually returns.

So if you find that your custom command always succeeds, even when it’s supposed to fail, check that you have the necessary call to os.Exit in TestMain.

Great. Now we can test that our program succeeds and fails when it should. What about more complicated behaviours, such as those involving command-line arguments?

For example, let’s extend our hello program to take a command-line argument, and fail if it’s not provided. Since all the real work is done in hello.Main, that’s where we need to make this change:

func Main() int {
    if len(os.Args[1:]) < 1 {
        fmt.Fprintln(os.Stderr, "usage: hello NAME")
        return 1
    }
    fmt.Println("Hello to you,", os.Args[1])
    return 0
}

(Listing hello/2)

This program now has two behaviours. When given an argument, it should print a greeting using that argument, and succeed. On the other hand, when the argument is missing, it should print an error message and fail.

Let’s test both behaviours in the same script:

# With no arguments, fail and print a usage message
! exec hello
! stdout .
stderr 'usage: hello NAME'

# With an argument, print a greeting using that value
exec hello Joumana
stdout 'Hello to you, Joumana'
! stderr .

(Listing hello/2)

The ability to define and run custom programs in this way is the key to using testscript to test command-line tools. We can invoke the program with whatever arguments, environment variables, and supporting files are required to test a given behaviour. In this way we can test even quite complex behaviours with a minimum of code.

And that’s no surprise, because testscript is derived directly from the code used to test the Go tool itself, which is probably as complex a command-line tool as any. It’s part of the very handy go-internal repository:

https://github.com/rogpeppe/go-internal

Checking the test coverage of scripts

One especially neat feature of testscript is that it can even provide us with coverage information when testing our binary. That’s something we’d find hard to do if we built and executed the binary ourselves, but testscript makes it work seemingly by magic:

go test -coverprofile=cover.out

PASS
coverage: 100.0% of statements

Since our hello script executes both of the two possible code paths in hello.Main, it covers it completely. Thus, 100% of statements.

Just to check that this is really being calculated properly, let’s try deliberately reducing the coverage, by testing only the happy path behaviour in our script:

# With an argument, print a greeting using that value
exec hello Joumana
stdout 'hello to you, Joumana'
! stderr .

We’re no longer causing the “if no arguments, error” code path to be executed, so we should see the total coverage go down:

coverage: 60.0% of statements

Since we’ve generated a coverage profile (the file cover.out), we can use this with the go tool cover command, or our IDE. This coverage profile can show us exactly which statements are and aren’t executed by tests (including test scripts). If there are important code paths we’re not currently covering, we can add or extend scripts so that they test those behaviours, too.

Test coverage isn’t always the most important guide to the quality of our tests, since it only proves that statements were executed, not what they do. But it’s very useful that we can test command-line tools and other programs as binaries using testscript, without losing our test coverage statistics.

Comparing output with files using cmp

Let’s look at some more sophisticated ways we can test input and output from command-line tools using testscript.

For example, suppose we want to compare the program’s output not against a string or regular expression, but against a prepared file that contains the exact output we expect. This is sometimes referred to as a golden file.

We can supply a golden file as part of the script file itself, delimiting its contents with a special marker line beginning and ending with a double hyphen (--).

Here’s an example:

exec hello
cmp stdout golden.txt

-- golden.txt --
hello world

(Listing hello/1)

The marker line containing golden.txt begins a file entry: everything following the marker line will be written to golden.txt and placed in the script’s work directory before it starts executing. We’ll have more to say about file entries later in this series, but first, let’s see what we can do with this file.

The cmp assertion can compare two files to see if they’re the same. If they match exactly, the test passes. If they don’t, the failure will be accompanied by a diff showing which parts didn’t match.

If the program’s output doesn’t match the golden file, as it won’t in this example, we’ll see a failure message like this:

> exec hello
[stdout]
hello world
> cmp stdout golden.txt
--- stdout
+++ golden.txt
@@ -1,1 +0,0 @@
-hello world
@@ -0,0 +1,1 @@
+goodbye world

FAIL: testdata/script/hello.txtar:2: stdout and golden.txt differ

Alternatively, we can use ! to negate the comparison, in which case the files must not match, and the test will fail if they do:

exec echo hello
! cmp stdout golden.txt

-- golden.txt --
goodbye world

(Listing hello/1)

The first argument to cmp can be the name of a file, but we can also use the special name stdout, meaning the standard output of the previous exec. Similarly, stderr refers to the standard error output.

If the program produces different output depending on the value of some environment variable, we can use the cmpenv assertion. This works like cmp, but interpolates environment variables in the golden file:

exec echo Running with home directory $HOME
cmpenv stdout golden.txt

-- golden.txt --
Running with home directory $HOME

(Listing hello/1)

When this script runs, the $HOME in the echo command will be expanded to the actual value of the HOME environment variable, whatever it is. But because we’re using cmpenv instead of cmp, we also expand the $HOME in the golden file to the same value.

So, assuming the command’s output is correct, the test will pass. This prevents our test from flaking when its behaviour depends on some environment variable that we don’t control, such as $HOME.

More matching: exists, grep, and -count

Some programs create files directly, without producing any output on the terminal. If we just want to assert that a given file exists as a result of running the program, without worrying about the file’s contents, we can use the exists assertion.

For example, suppose we have some program myprog that writes its output to a file specified by the -o flag. We can check for the existence of that file after running the program using exists:

exec myprog -o results.txt
exists results.txt

And if we are concerned about the exact contents of the results file, we can use cmp to compare it against a golden file:

exec myprog -o results.txt
cmp results.txt golden.txt

-- golden.txt --
hello world

If the two files match exactly, the assertion succeeds, but otherwise it will fail and produce a diff showing the mismatch. If the results file doesn’t exist at all, that’s also a failure.

On the other hand, if we don’t need to match the entire file, but only part of it, we can use the grep assertion to match a regular expression:

exec myprog -o results.txt
grep '^hello' results.txt

-- golden.txt --
hello world

A grep assertion succeeds if the file matches the given expression at least once, regardless of how many matches there are. On the other hand, if it’s important that there are a specific number of matches, we can use the -count flag to specify how many :

grep -count=1 'beep' result.txt

-- result.txt --
beep beep

In this example, we specified that the pattern beep should only match once in the target file, so this will fail:

> grep -count=1 'beep' result.txt
[result.txt]
beep beep

FAIL: beep.txtar:1: have 2 matches for `beep`, want 1

Because the script’s work directory is automatically deleted after the test, we can’t look at its contents—for example, to figure out why the program’s not behaving as expected. To keep this directory around for troubleshooting, we can supply the -testwork flag to go test.

This will preserve the script’s work directory intact, and also print the script’s environment, including the WORK variable that tells us where to find that directory:

--- FAIL: Test/hello (0.01s)
    testscript.go:422:
        WORK=/private/var/folders/.../script-hello
        PATH=...
        ...

That’s it for Part 2; in Part 3, we’ll find out more about this mysterious txtar format, and we’ll also learn how to supply standard input to programs running in test scripts. Don’t miss it!

Previous: Test scripts in Go

Next: Files in test scripts