Fuzz tests in Go

From The Power of Go: Tests

Thinking up test cases is hard. Let’s not bother.

Fuzz testing works by throwing everything at the wall and seeing what sticks.

—Xuanyi Chew, “Property-Based Testing”

This is the second of four tutorials on Go fuzz tests:

1. Random testing in Go
2. Fuzz tests in Go
3. Writing a Go fuzz target
4. Finding bugs with fuzzing

Nobody’s perfect, especially programmers, and while we try to be reasonably careful when writing code, we’re bound to overlook things from time to time.

Bugs often arise when the programmer doesn’t envision certain kinds of inputs, and thus doesn’t test that the system handles them correctly. In other words, you can only find bugs you can think of.

Slice overruns are a classic example of the kind of bug that busy programmers can easily overlook:

results := SearchProducts(productID)
return results[0].Name // panics if slice is empty or nil

Indeed, this code might run perfectly well in production for months or even years, and then suddenly break when someone searches for a product that doesn’t exist. It wasn’t caught by any example-based tests because nobody thought about that situation.

But there’s another kind of testing that can help: fuzz testing. The “fuzz” here refers not to data that’s blurry, or imprecise, but means simply “randomly generated”. We saw in Part 1 that it’s possible, and useful, to generate random test inputs in plain old Go code. But it’s nice to have an automated way of doing it, too.

Tools for fuzz testing have been around for decades, but built-in support for it was added to Go relatively recently, in version 1.18.

Could we boost our chances of finding lurking bugs in a Go function by fuzzing it—that is, by randomly generating lots of different inputs, and seeing what happens?

To find out, let’s deliberately write a function that will panic when given a certain secret trigger value:

func Guess(n int) {
    if n == 21 {
        panic("blackjack!")
    }
}

(Listing guess/1)

Now, we’ll pretend we don’t know the special value, but we merely suspect that such a bug lurks within the function. We’ll try to find it using randomised inputs. How could we do that?

We need to generate a large number of random values that decently cover the space of likely inputs, and call the function with each of them in turn. If one of them causes a panic, then we have found our bug.

It’s not that we couldn’t write this by hand, as in the previous examples, but instead let’s try writing a fuzz test, using Go’s standard testing package.

Here’s what that looks like:

func FuzzGuess(f *testing.F) {
    f.Fuzz(func(t *testing.T, input int) {
        guess.Guess(input)
    })
}

(Listing guess/1)

The names of fuzz tests in Go start with the word Fuzz, just as regular tests start with the word Test. And the parameter they take is a *testing.F, which is similar to our friend the *testing.T, and has many of the same methods.

The fuzz target

What do we do with this f parameter, then? We call its Fuzz method, passing it a certain function. Passing functions to functions is very Go-like, and though it can be confusing when you’re new to the idea, it’s also very powerful once you grasp it.

So what’s this function within a function? In fact, it’s the function that the fuzz testing machinery (the fuzzer, for short) will repeatedly call with the different values it generates.

We call this function the fuzz target, and here is its signature:

func(t *testing.T, input int)

Remember, this isn’t the test function, it’s the function we pass to f.Fuzz. And although it must take a *testing.T as its first argument, it can also take any number of other arguments. In this example, it takes a single int argument that we’ll call input.

These extra arguments to the fuzz target represent your inputs, which the fuzzer is going to randomly generate. In other words, the fuzzer’s job is to call this function—the fuzz target—with lots of different values for input, and see what happens.

And here’s the fuzz target we’ll be using in this example:

func(t *testing.T, input int) {
    guess.Guess(input)
}

This very simple fuzz target does nothing but call Guess with the generated input. We could do more here; indeed, we could do anything that a regular test can do using the t parameter.

But to keep things simple for now, we’ll just call the function under test, without actually checking anything about the result. So the only way this fuzz target can fail is if the function either panics, or takes so long to return that the test times out.

Running tests in fuzzing mode

To start fuzzing, we use the go test command, adding the -fuzz flag:

go test -fuzz .

Note the dot after the -fuzz, which is significant. Just as the -run flag takes a regular expression to specify which tests to run, the -fuzz flag does the same. In this case, we’re using the regular expression “.”, which matches all fuzz tests.

Here’s the result:

warning: starting with empty corpus
fuzz: elapsed: 0s, execs: 0 (0/sec), new interesting: 0 (total: 0)
--- FAIL: FuzzGuess (0.05s)
    --- FAIL: FuzzGuess (0.00s)
        testing.go:1356: panic: blackjack!
        [stack trace omitted]

Failing input written to testdata/fuzz/FuzzGuess/xxx
    To re-run:
    go test -run=FuzzGuess/xxx

The first thing to note about this output is that the test failed. Specifically, it failed with this panic:

panic: blackjack!

This is the panic message that we deliberately hid inside the Guess function. It didn’t take too long for the fuzzer to find the “secret” input that triggers the panic. But what was it?

This isn’t shown as part of the output, but there’s a clue:

Failing input written to testdata/fuzz/FuzzGuess/xxx

Instead of xxx, the actual filename is some long hexadecimal string that I won’t weary you by reproducing in full. We don’t normally need to look at the contents of these files, but just for fun, let’s take a peek:

go test fuzz v1
int(21)

The first line identifies what kind of file this is: it’s generated by v1 of the Go fuzzer. Including a version identifier like this is a good idea when generating any kind of machine-readable file, by the way. If you decide you’d like to change the file format later, you can alter the version number to maintain compatibility with older versions of the reading program.

The second line contains the information we’re really interested in:

int(21)

This gives the specific input value represented by this test case. It’s an int, which is no surprise, since that’s what our fuzz target takes, and the value that triggered a failure is the magic 21. Winner, winner, chicken dinner!

Failing inputs become static test cases

Fuzz testing is great at discovering unexpected inputs that expose weird behaviour.

—Jay Conrod, “Internals of Go’s new fuzzing system”

Why has the fuzzer generated this testdata file, then? Why not just report the failing input as part of the test results, for example?

Well, suppose we grow bored of fuzzing for the time being, and we decide to just run our plain old tests again, without the -fuzz flag:

go test

--- FAIL: FuzzGuess (0.00s)
    --- FAIL: FuzzGuess/xxx (0.00s)
panic: blackjack! [recovered]

Wait, what? Why is it running the fuzz test, if we didn’t say -fuzz?

The answer, as you may have guessed, is that fuzz tests can also act as regular tests. When run in non-fuzzing mode (that is, without the -fuzz flag), they don’t generate any random inputs. What they do instead is call the fuzz target with any and all inputs that have previously been found to trigger a failure. These are the inputs stored in the special files under testdata/fuzz.

As it happens, there’s only one such input at the moment (21), but it’s enough to fail the test. A good way to think about the fuzzer is that its job is to generate failing test cases, which then become part of your normal example-based tests.

In other words, once some randomly-generated failing input has been found by the fuzzer, running go test will test your function with that input ever afterwards, unless you choose to remove the testdata file manually.

Let’s fix the function so that it no longer panics on 21, or indeed does anything at all:

func Guess(n int) {}

(Listing guess/2)

Let’s use go test with the -v flag to show that the 21 test case now passes:

go test -v

=== RUN   FuzzGuess
=== RUN   FuzzGuess/xxx
--- PASS: FuzzGuess (0.00s)
    --- PASS: FuzzGuess/xxx (0.00s)

This is a neat feature, as it means that the fuzzer automatically creates regression tests for us. If we should ever write a bug in the future that would cause the function to fail given this same input, go test will catch it. Every time a new failing test case is found by the fuzzer, it becomes part of our regular test suite.

In Part 3, we’ll see how to write a more sophisticated fuzz target, to track down a more sophisticated bug. Also, cake will be served, so don’t miss it.