Testing errors in Go

From The Power of Go: Tests

While it is a known fact that programmers never make mistakes, it is still a good idea to humor the users by checking for errors at critical points in your programs.

—Robert D. Schneider, “Optimizing INFORMIX Applications”

This is the first of a three-part series of tutorials on Golang error testing:

1. Testing errors in Go
2. Comparing Go error values
3. Error wrapping in Go

Ignoring errors is a mistake

First, let’s talk about unexpected errors in Go tests. It’s common for Go functions to return an error value as part of their results. It’s especially common for functions to return “something and error”:

func CreateUser(u User) (UserID, error) {

How do we test a function like this? Let’s look at a few of the wrong ways to do it first.

One really wrong way would be something like this:

func TestCreateUser(t *testing.T) {
    want := 1
    got, _ := CreateUser("some valid user")
    if want != got {
        t.Errorf("want user ID %d, got %d", want, got)
    }
}

We know from its signature that CreateUser returns an error as part of its API, but we ignore this result value completely in the test, by assigning it to the blank identifier (_).

And you can see why this is a bad idea, can’t you? It’s another example of the happy-path fixation. If there were some bug that caused CreateUser to return an error when it shouldn’t, would this test detect it? Nope.

But there’s a more subtle problem, too. What if there were a bug in the test? We appear to be testing the “valid input” behaviour, but what if we mistakenly passed invalid input instead? What would we see when we ran the test?

want user ID 1, got 0

Wait, what? We can stare at the code for CreateUser as long as we like, and we won’t see the problem that’s causing this, because it isn’t there.

The problem is not that want doesn’t equal got, although that happens to be true. The problem is that we shouldn’t have even looked at got, because there was an error. By returning a zero user ID and an error, CreateUser is trying to tell us that something’s gone wrong, but we’re not listening.

This is something that often trips up people new to Go. In some languages, functions can signal an unhandled error condition using an exception, for example, which would cause a test like this to fail automatically. This isn’t the case in Go, where we signal that got is invalid by also returning a non-nil error.

That puts the burden of checking the error on the caller: in this case, that’s the test. So, as we can see, it’s easy to make a mistake where an unchecked error results in the code proceeding as though got were valid when it’s not.

We don’t want to write tests for our tests (where would it end?), but we can write our tests in a defensive way, by always checking errors. At least then if we do make a mistake, the resulting failure will give us a clue about what it was.

Ignoring error values in a test, or indeed anywhere in a Go program, is a pretty common mistake. It might save us a second or two now, but it’ll cost us (or our successors) a lot of puzzlement later. Let’s not store up any more trouble for the future than we have to.

A great way to add value and robustness to any existing Go test suite is to go through it looking for places where errors are ignored using _ (static analysers such as errcheck can find such places for you). Remove the blank identifier and assign the error result to the err variable, then check it and fail the test with t.Fatal if necessary.

Unexpected errors should stop the test

How exactly should we fail the test if there’s an unexpected error? One idea is to call t.Error, but is that good enough?

func TestCreateUser(t *testing.T) {
    ...
    got, err := CreateUser(testUser)
    if err != nil {
        t.Error(err)
    }
    ... // more testing here
}

No, because t.Error, though it marks the test as failed, also continues the test. That’s not the right thing to do here. We need to stop the test right away. Why?

If err is not nil, then we don’t have a valid result, so we shouldn’t go on to test anything about it. Indeed, even looking at the value of got could be dangerous.

Consider, for example, some function store.Open:

func Open(path string) (*Store, error) {
    ... // load data from 'path' if possible
    ... // but this could fail:
    if err != nil {
        return nil, err
    }
    return &Store{
        Data: data,
    }, nil
}

It’s conventional among Gophers that functions like this should return a nil pointer in the error case. So any code that tries to dereference that pointer will panic when it’s nil:

func TestOpen(t *testing.T) {
    s, err := store.Open("testdata/store.bin")
    if err != nil {
        t.Error(err)
    }
    for _, v := range s.Data { // no! panics if s is nil
        ...
    }
}

Suppose store.bin doesn’t exist, so Open can’t open it, and returns an error. The test detects this, and calls t.Error to report it, but then continues.

Now we’re going to try to range over s.Data. But that won’t work when s is nil, as it will be if there was an error. Dereferencing the s pointer will cause the test to panic, which is confusing.

Instead, the test should call t.Fatalf as soon as it detects an error from store.Open:

s, err := store.Open("testdata/store.bin")
if err != nil {
    t.Fatalf("unexpected error opening test store: %v", err)
}

Error behaviour is part of your API

So that’s how we deal with unexpected errors, but what about expected errors? After all, if errors are part of our public API, which they usually are, then we need to test them. Arguably, the way the system behaves when things go wrong is even more important than what it does when things go right.

At a minimum, our tests should check two things. One, that the system produces an error when it should. Two, that it doesn’t produce one when it shouldn’t.

Let’s apply this idea to testing some function format.Data, for example. Here’s the “want error” test:

func TestFormatData_ErrorsOnInvalidInput(t *testing.T) {
    _, err := format.Data(invalidInput)
    if err == nil {
        t.Error("want error for invalid input")
    }
}

(Listing format/1)

For the “want no error” case, we need to check not only that there’s no error, but that we also get the expected result:

func TestFormatData_IsCorrectForValidInput(t *testing.T) {
    want := validInputFormatted
    got, err := format.Data(validInput)
    if err != nil {
        t.Fatal(err)
    }
    if want != got {
        t.Error(cmp.Diff(want, got))
    }
}

(Listing format/1)

In Part 2, we’ll talk about how to produce certain errors in order to test them, and also about how to compare error values in Go. See you soon!