Standalone test scripts

From The Power of Go: Tests

In this final part of our series on test scripts in Go, we’ll talk about how to run test scripts not in Go. That is, test scripts as standalone programs, using the nifty testscript command.

The standalone testscript runner

The testscript language is so useful that it would be great to be able to use it even outside the context of Go tests. For example, we might like to write test scripts as standalone programs for use in automation pipelines and CI systems, or in non-Go software projects.

Wouldn’t it be nice if we could write a test script and simply run it directly from the command line, without having to write Go code to do so? Well, you’ll be delighted to know that we can do exactly this, by installing the standalone testscript tool:

go install github.com/rogpeppe/go-internal/cmd/testscript@latest

To use it, all we need to do is give the path to a script, or multiple scripts:

testscript testdata/script/*

This will run each script in turn and print PASS if it passes (and the comments describing each successful phase). Otherwise, it will print FAIL, accompanied by the same failure output we’d see when running the script in a Go test.

If any script fails, the exit status from testscript will be 1, which is useful for detecting failures in automations.

If we want to log what the script is doing, we can use the -v flag, which prints verbose output whether the script passes or fails:

testscript -v echo.txtar

WORK=$WORK
[rest of environment omitted]

> exec echo hello
[stdout]
hello
> stdout 'hello'
PASS

To pass environment variables to scripts, we can specify them using the -e flag. Repeat the -e flag for each variable-value pair:

testscript -e VAR1=hello -e VAR2=goodbye script.txtar

Just as when running scripts from Go tests, each script will get its own work directory which is cleaned up afterwards. To preserve this directory and its contents, for example for troubleshooting, use the -work flag:

testscript -work script.txtar

temporary work directory: /var/.../testscript1116180846
PASS

By the way, if you use VS Code, there’s a useful extension available for syntax highlighting txtar files and scripts, called vscode-txtar. Not only does it highlight testscript syntax, it’s also smart enough (in most cases) to identify the language in included files (for example, .go files) and highlight them appropriately. This makes editing non-trivial scripts a good deal easier.

Just for fun, we can even use testscript as an interpreter, using the “shebang line” syntax available on some Unix-like operating systems, including Linux and macOS.

For example, we could create a file named hello.txtar with the following contents:

#!/usr/bin/env testscript
exec echo hello
stdout 'hello'

The line beginning #! tells the system where to find the interpreter that should be used to execute the rest of the script. If you’ve wondered why some shell scripts start with #!/bin/sh, for example, now you know.

So, if we alter the file’s permissions to make it executable, we can run this script directly from the command line:

chmod +x hello.txtar

./hello.txtar

PASS

Test scripts as issue repros

Because the flexible txtar format lets us represent not only a test script, but also multiple files and folders as a single copy-pastable block of text, it’s a great way to submit test case information along with bug reports. Indeed, it’s often used to report bugs in Go itself.

If we have found some bug in a program, for example, we can open an issue with the maintainers and provide them with a test script that very concisely demonstrates the problem:

# I was promised 'Go 2', are we there yet?
exec go version
stdout 'go version go2.\d+'

The maintainers can then run this script themselves to see if they can reproduce the issue:

testscript repro.txtar

# I was promised 'Go 2', are we there yet? (0.016s)
> exec go version
[stdout]
go version go1.18 darwin/amd64

> stdout 'go version go2.\d+'
FAIL: script.txt:3: no match for `go version go2.\d+` found in stdout

Supposing they’re willing or able to fix the bug (if indeed it is a bug), they can check their proposed fix using the same script:

testscript repro.txtar

# I was promised 'Go 2', are we there yet? (0.019s)
PASS

Indeed, they can add the script to the project’s own tests. By making it easy to submit, reproduce, and discuss test cases, testscript can even be a way to gently introduce automated testing into an environment where it’s not yet seriously practiced:

How do you spread the use of automated testing? One way is to start asking for explanations in terms of test cases: “Let me see if I understand what you’re saying. For example, if I have a Foo like this and a Bar like that, then the answer should be 76?”

—Kent Beck, “Test-Driven Development by Example”

Test scripts as… tests

Didn’t we already talk about this? We started out in Part 1 describing testscript as an extension to Go tests. In other words, running a script via the go test command. But what if you want to flip the script again, by running the go test command from a script?

I’ll give you an example. I maintain all the code examples from The Power of Go: Tests and other books in GitHub repos. I’d like to check, every time I make changes, that all the code still behaves as expected. How can I do that?

I could manually run go test in every module subdirectory, but there are dozens in the tpg-tests repo alone, so this would take a while. I could even write a shell script that visits every directory and runs go test, and for a while that’s what I did. But it’s not quite good enough.

Because I’m developing the example programs step by step, some of the example tests don’t pass, and that’s by design. For example, if I start by writing a failing test for some feature, in the “guided by tests” style I prefer, then I expect it to fail, and that’s what my script needs to check.

How can I specify that the go test command should fail in some directories, but pass in others? That sounds just like the exec assertion we learned about in Part 1, doesn’t it? And we can use exactly that:

! exec go test
stdout 'want 4, got 5'

(Listing double/1)

You can see what this is saying: it’s saying the tests should fail, and the failure output should contain the message “want 4, got 5”. And that’s exactly what the code does, so this script test passes. It might seem a little bizarre to have a test that passes if and only if another test fails, but in this case that’s just what we want.

Many of the examples are supposed to pass, of course, so their test scripts specify that instead:

exec go test

(Listing double/2)

And in most cases that’s the only assertion necessary. But there’s something missing from these scripts that you may already have noticed. Where’s the code?

We talked in Part 3 about the neat feature of the txtar format that lets us include arbitrary text files in a test script. These will automatically be created in the script’s work directory before it runs. But we don’t see any Go files included in these test scripts, so where do they come from? What code is the go test command testing?

Well, it’s in the GitHub repo, of course: Listing double/2, for example. I don’t want to have to copy and paste it all into the test script, and even if I did, that wouldn’t help. The test script is supposed to tell me if the code in the repo is broken, not the second-hand copy of it I pasted into the script.

We need a way of including files in a script on the fly, so to speak. In other words, what we’d like to do is take some directory containing a bunch of files, bundle them all up in txtar format, add some test script code, and then run that script with testscript. Sounds complicated!

Not really. The txtar-c tool does exactly this. Let’s install it:

go install github.com/bitfield/txtar-c@latest

Now we can create a txtar archive of any directory we like:

txtar-c .

The output is exactly what you’d expect: a txtar file that includes the contents of the current directory. Great! But we also wanted to include our exec assertion, so let’s put that in a file called test.txtar, and have the tool automatically include it as a script:

txtar-c -script test.txtar .

We now have exactly the script that we need to run, and since we don’t need to save it as a file, we can simply pipe it straight into testscript:

txtar-c -script test.txtar . |testscript

PASS

Very handy! So txtar-c and testscript make perfect partners when we want to run some arbitrary test script over a bunch of existing files, or even dozens of directories containing thousands of files. When it’s not convenient to create the txtar archive manually, we can use txtar-c to do it automatically, and then use the standalone testscript runner to test the result.

Conclusion

To sum up, test scripts are a very Go-like solution to the problem of testing command-line tools: they can do a lot with a little, using a simple syntax that conceals some powerful machinery.

The real value of testscript, indeed, is that it strips away all the irrelevant boilerplate code needed to build binaries, execute them, read their output streams, compare the output with expectations, and so on.

Instead, it provides an elegant and concise notation for describing how our programs should behave under various different conditions. By relieving our brains of this unnecessary work, testscript sets us free to think about more advanced problems, such as: what are we really testing here?

Previous: Conditions and concurrency