Conditions and concurrency

From The Power of Go: Tests

In Part 1 of this series, we got started with the testscript package. In Part 2 we learned how to test Go CLI tools using a test script, and in Part 3 we explored the txtar format and ways to manipulate files and folders.

Let’s look at some advanced testscript techniques now, including conditions, environment variables, and concurrency.

Conditions

Because we’re running real programs on a real computer that we have limited information about, we may need to specify some conditions governing whether or not to perform an action.

For example, we can’t always guarantee in advance that the program we want to run will actually be available on the test system. We can check for its existence by prefixing the script line with a condition:

[exec:sh] exec echo yay, we have a shell

The square brackets contain a condition, which can be true or false. If the condition is true, the rest of the line is executed. Otherwise, it’s ignored.

In this example, the condition [exec:sh] is true if there is a program named sh in our $PATH that we have permission to execute. If there isn’t, this line of the script will be ignored.

On the other hand, if we need to check for a program at a specific path, we can give that path as part of the condition:

[exec:/bin/sh] exec echo yay, we have /bin/sh

There are a few other built-in conditions available, but perhaps the most useful to us are the current Go version, the current operating system, and the CPU architecture of this machine:

# 'go1.x' is true if this is Go 1.x or higher
[go1.16] exec echo 'We have at least Go 1.16'

# Any known value of GOOS is also a valid condition
[darwin] exec echo 'We''re on macOS'

# As is any known value of GOARCH
[!arm64] exec echo 'This is a non-arm64 machine'

As you can see from that example, we can also negate a condition by prefixing it with !. For example, the condition [!arm64] is true only if the value of GOARCH is not arm64.

We can use a condition to control other script statements, too, not just exec. For example, we can use the skip statement to skip the test in some circumstances:

# Skip this test unless we have Go 1.18 or later
[!go1.18] skip

# Skip this test on Linux
[linux] skip

The unix condition is true when the target OS is one of those that Go considers “Unix-like”, including Linux, macOS, FreeBSD, and others.

[unix] exec echo 'It''s a UNIX system! I know this!'

In Go 1.19, the only supported operating systems for which unix is not true are js, nacl, plan9, windows, and zos.

Setting environment variables with env

Since command-line tools often use environment variables, it’s important to be able to set and use these in scripts. To set a particular variable for the duration of the script, use an env statement:

env MYVAR=hello

For example, to test that some program myprog fails and prints an error message when the environment variable it needs has no value, we could write:

env AUTH_TOKEN=
! exec myprog
stderr 'AUTH_TOKEN must be set'

In scripts, we can refer to the value of an environment variable by using the $ token followed by the name of the variable, just like in a shell script. This will be replaced by the value of the variable when the script runs. For example:

exec echo $PATH

This will cause echo to print the contents of $PATH, whatever it happens to be when the script is run.

Each script starts with an empty environment, apart from $PATH, and a few other predefined variables. For example, HOME is set to /no-home, because many programs expect to be able to find the user’s home directory in $HOME.

If scripts need to know the absolute path of their work directory, it will be available as $WORK. For example, we could use $WORK as part of the value for another environment variable:

env CACHE_DIR=$WORK/.cache

To make scripts deterministic, though, the actual value of this variable will be replaced in the testscript output by the literal string $WORK. In fact, the value of $WORK will be different for every run and every test, but it will always print as simply $WORK.

Because some programs also expect to read the variable $TMPDIR to find out where to write their temporary files, testscript sets this too, and its value will be $WORK/.tmp.

A useful variable for cross-platform testing is $exe. Its value on Windows is .exe, but on all other platforms, it’s the empty string. So you can specify the name of some program prog like this:

exec prog$exe

On Windows, this evaluates to running prog.exe, but on non-Windows systems, it will be simply prog.

Passing values to scripts via environment variables

Sometimes we need to supply dynamic information to a script, for example some value that’s not known until the test is actually running. We can pass values like this to scripts using the environment, by using the Setup function supplied to testscript.Run as part of Params.

For example, suppose the program run by the script needs to connect to some server, but the required port number won’t be known until we start the server in the test. Let’s say it’s generated randomly.

Once we know the generated address, we can store it in a suitable environment variable, using the Setup function supplied to testscript.Run:

func TestScriptWithExtraEnvVars(t *testing.T) {
    t.Parallel()
    addr := randomLocalAddr(t)
    testscript.Run(t, testscript.Params{
        Dir: "testdata/script",
        Setup: func(env *testscript.Env) error {
            env.Setenv("SERVER_ADDR", addr)
            return nil
        },
    })
}

(Listing env/1)

The Setup function is run after any files in the test script have been extracted, but before the script itself starts. It receives an Env object containing the predefined environment variables to be supplied to the script.

At this point, our Setup function can call env.Setenv to add any extra environment variables it wants to make available in the script. For example, we can set the SERVER_ADDR variable to whatever the test server’s address is.

A script can then get access to that value by referencing $SERVER_ADDR, just as it would with any other environment variable:

exec echo $SERVER_ADDR
stdout '127.0.0.1:[\d]+'

(Listing env/1)

The stdout assertion here is just to make sure that we did successfully set the SERVER_ADDR variable to something plausible. In a real test we could use this value as the address for the program to connect to.

Running programs in background with &

One nice feature of typical shells is the ability to run programs in the background: that is, concurrently. For example, in most shells we can write something like:

sleep 10 &

This starts the sleep program running as a separate process, but continues to the next line of the script so that we can do something else concurrently. We can do the same in a test script:

exec sleep 10 &

The trailing & (“ampersand”) character tells testscript to execute the sleep program in the background, but carry on. Its output will be buffered so that we can look at it later, if we need to. We can start as many programs in the background as necessary, and they will all run concurrently with the rest of the script.

Because backgrounding a program with & doesn’t wait for it to complete before continuing, we need a way to do that explicitly. Otherwise, the script might end before the background program is done. To do that, we use the wait statement:

exec sleep 10 &
wait

This pauses the script until all backgrounded programs have exited, and then continues. At this point the output from those programs, in the order they were started, will be available for testing using the stdout, stderr, and cmp assertions, just as if they’d been run in the foreground.

Why might we want to do this? Well, some programs might need some background service to be available in order for us to test them, such as a database or a web server.

At the end of the script, all programs still running in background will be stopped using the os.Interrupt signal (that is, the equivalent of typing Ctrl-C at the terminal), or, if necessary, os.Kill.

Let’s see an example. Suppose we have some Go program that implements a web server, and we want to start it on some arbitrary port, then connect to it with curl and check that we get the expected welcome message. How could we test that?

First, let’s set up a delegate Main function to run the server program, taking its listen address from os.Args. We’ll associate this with a custom program named listen, using testscript.RunMain in the same way that we’ve done for previous examples.

Again, we’ll generate a random listen address, and put this value in the environment variable SERVER_ADDR, using Setup. The details are in Listing env/2, but they’re essentially the same as in the previous examples.

Now we can write the script itself:

exec listen $SERVER_ADDR &
exec curl -s --retry-connrefused --retry 1 $SERVER_ADDR
stdout 'Hello from the Go web server'
wait

(Listing env/2)

The first thing we do is use exec to start our listen program on the generated address. Notice that we use the & token to execute the program concurrently, in background.

The next line of the script uses the well-known curl tool to make an HTTP connection to the given address, and print the response. Because it takes some non-zero time for our server program to actually start, if we try to connect instantly it probably won’t work. Instead, we need to tell curl to retry if the first connection attempt is refused.

Once the connection has succeeded, we use stdout to assert that the server responds with the message “Hello from the Go web server”.

If the script simply ended here, the background listen process would be killed with an interrupt signal, which would cause the test to fail. We don’t want that, so we add a wait statement.

But if the server listened forever, then the wait statement would also wait for it forever (or, at least, until the test timed out). We don’t want that either, so we’ve arranged for the server to politely shut itself down after serving exactly one request. Normal servers wouldn’t do this, but it just makes this demonstration clearer.

Therefore, the wait statement pauses the script just long enough to allow this tidy shutdown to happen, and then exits.

Thanks for reading Part 4. In Part 5 we’ll talk about standalone test scripts and the testscript CLI tool.

Previous: Files in test scripts

Next: Standalone test scripts