Standalone test scripts
In this final part of our series on test scripts in Go, we’ll talk
about how to run test scripts not in Go. That is, test scripts
as standalone programs, using the nifty testscript
command.
The standalone
testscript
runner
The testscript
language is so useful that it would be
great to be able to use it even outside the context of Go
tests. For example, we might like to write test scripts as standalone
programs for use in automation pipelines and CI systems, or in non-Go
software projects.
Wouldn’t it be nice if we could write a test script and simply run it
directly from the command line, without having to write Go code to do
so? Well, you’ll be delighted to know that we can do exactly this, by
installing the standalone testscript
tool:
go install github.com/rogpeppe/go-internal/cmd/testscript@latest
To use it, all we need to do is give the path to a script, or multiple scripts:
testscript testdata/script/*
This will run each script in turn and print PASS
if it
passes (and the comments describing each successful phase). Otherwise,
it will print FAIL
, accompanied by the same failure output
we’d see when running the script in a Go test.
If any script fails, the exit status from testscript
will be 1, which is useful for detecting failures in automations.
If we want to log what the script is doing, we can use the
-v
flag, which prints verbose output whether the script
passes or fails:
testscript -v echo.txtar
WORK=$WORK
[rest of environment omitted]
> exec echo hello
[stdout]
hello
> stdout 'hello'
PASS
To pass environment
variables to scripts, we can specify them using the -e
flag. Repeat the -e
flag for each variable-value pair:
testscript -e VAR1=hello -e VAR2=goodbye script.txtar
Just as when running scripts from Go
tests, each script will get its own work directory which is cleaned
up afterwards. To preserve this directory and its contents, for example
for troubleshooting, use the -work
flag:
testscript -work script.txtar
temporary work directory: /var/.../testscript1116180846
PASS
By the way, if you use VS Code,
there’s a useful extension available for syntax highlighting
txtar
files and scripts, called vscode-txtar
.
Not only does it highlight testscript
syntax, it’s also
smart enough (in most cases) to identify the language in included files
(for example, .go
files) and highlight them
appropriately. This makes editing non-trivial scripts a good deal
easier.
Just for fun, we can even use testscript
as an
interpreter, using the “shebang line”
syntax available on some Unix-like operating systems, including Linux
and macOS.
For example, we could create a file named hello.txtar
with the following contents:
#!/usr/bin/env testscript
exec echo hello
stdout 'hello'
The line beginning #!
tells the system where to find the
interpreter that should be used to execute the rest of the script. If
you’ve wondered why some shell scripts start with
#!/bin/sh
, for example, now you know.
So, if we alter the file’s permissions to make it executable, we can run this script directly from the command line:
chmod +x hello.txtar
./hello.txtar
PASS
Test scripts as issue repros
Because the flexible txtar
format lets us represent not
only a test script, but also multiple files and folders as a single
copy-pastable block of text, it’s a great way to submit test case
information along with bug reports. Indeed, it’s often used to report bugs in Go
itself.
If we have found some bug in a program, for example, we can open an issue with the maintainers and provide them with a test script that very concisely demonstrates the problem:
# I was promised 'Go 2', are we there yet?
exec go version
stdout 'go version go2.\d+'
The maintainers can then run this script themselves to see if they can reproduce the issue:
testscript repro.txtar
# I was promised 'Go 2', are we there yet? (0.016s)
> exec go version
[stdout]
go version go1.18 darwin/amd64
> stdout 'go version go2.\d+'
FAIL: script.txt:3: no match for `go version go2.\d+` found in stdout
Supposing they’re willing or able to fix the bug (if indeed it is a bug), they can check their proposed fix using the same script:
testscript repro.txtar
# I was promised 'Go 2', are we there yet? (0.019s)
PASS
Indeed, they can add the script to the project’s own tests. By making
it easy to submit, reproduce, and discuss test cases,
testscript
can even be a way to gently introduce automated
testing into an environment where it’s not yet seriously practiced:
How do you spread the use of automated testing? One way is to start asking for explanations in terms of test cases: “Let me see if I understand what you’re saying. For example, if I have a Foo like this and a Bar like that, then the answer should be 76?”
—Kent Beck, “Test-Driven Development by Example”
Test scripts as… tests
Didn’t we already talk about this? We started out in Part 1 describing testscript
as an extension to Go tests. In other words, running a script via the
go test
command. But what if you want to flip the script
again, by running the go test
command from a
script?
I’ll give you an example. I maintain all the code examples from The Power of Go: Tests and other books in GitHub repos. I’d like to check, every time I make changes, that all the code still behaves as expected. How can I do that?
I could manually run go test
in every module
subdirectory, but there are dozens in the tpg-tests repo alone,
so this would take a while. I could even write a shell script that
visits every directory and runs go test
, and for a while
that’s what I did. But it’s not quite good enough.
Because I’m developing the example programs step by step, some of the example tests don’t pass, and that’s by design. For example, if I start by writing a failing test for some feature, in the “guided by tests” style I prefer, then I expect it to fail, and that’s what my script needs to check.
How can I specify that the go test
command should fail
in some directories, but pass in others? That sounds just like the
exec
assertion we learned about in Part 1, doesn’t it? And we can use
exactly that:
! exec go test
stdout 'want 4, got 5'
You can see what this is saying: it’s saying the tests should fail, and the failure output should contain the message “want 4, got 5”. And that’s exactly what the code does, so this script test passes. It might seem a little bizarre to have a test that passes if and only if another test fails, but in this case that’s just what we want.
Many of the examples are supposed to pass, of course, so their test scripts specify that instead:
exec go test
And in most cases that’s the only assertion necessary. But there’s something missing from these scripts that you may already have noticed. Where’s the code?
We talked in Part 3 about the
neat feature of the txtar
format that lets us include
arbitrary text files in a test script. These will automatically be
created in the script’s work directory before it runs. But we don’t
see any Go files included in these test scripts, so where do
they come from? What code is the go test
command
testing?
Well, it’s in the GitHub repo, of course: Listing double/2
,
for example. I don’t want to have to copy and paste it all into the test
script, and even if I did, that wouldn’t help. The test script is
supposed to tell me if the code in the repo is broken, not the
second-hand copy of it I pasted into the script.
We need a way of including files in a script on the fly, so to speak.
In other words, what we’d like to do is take some directory containing a
bunch of files, bundle them all up in txtar
format, add
some test script code, and then run that script with
testscript
. Sounds complicated!
Not really. The txtar-c
tool
does exactly this. Let’s install it:
go install github.com/bitfield/txtar-c@latest
Now we can create a txtar
archive of any directory we
like:
txtar-c .
The output is exactly what you’d expect: a txtar
file
that includes the contents of the current directory. Great! But we also
wanted to include our exec
assertion, so let’s put that in
a file called test.txtar
, and have the tool automatically
include it as a script:
txtar-c -script test.txtar .
We now have exactly the script that we need to run, and since we
don’t need to save it as a file, we can simply pipe it straight into
testscript
:
txtar-c -script test.txtar . |testscript
PASS
Very handy! So txtar-c
and testscript
make
perfect partners when we want to run some arbitrary test script over a
bunch of existing files, or even dozens of directories
containing thousands of files. When it’s not convenient to create the
txtar
archive manually, we can use txtar-c
to
do it automatically, and then use the standalone testscript
runner to test the result.
Conclusion
To sum up, test scripts are a very Go-like solution to the problem of testing command-line tools: they can do a lot with a little, using a simple syntax that conceals some powerful machinery.
The real value of testscript
, indeed, is that it strips
away all the irrelevant boilerplate code needed to build binaries,
execute them, read their output streams, compare the output with
expectations, and so on.
Instead, it provides an elegant and concise notation for describing
how our programs should behave under various different conditions. By
relieving our brains of this unnecessary work, testscript
sets us free to think about more advanced problems, such as: what are we really testing here?
Previous: Conditions and concurrency