Comparing Go error values

From The Power of Go: Tests

I believe you should never inspect the output of the Error method. The Error method on the error interface exists for humans, not code. The contents of that string belong in a log file, or displayed on screen.

Comparing the string form of an error is, in my opinion, a code smell, and you should try to avoid it.
—Dave Cheney, “Don’t just check errors, handle them gracefully”

This is Part 2 of a series about errors and error testing in Go:

1. Testing errors in Go
2. Comparing Go error values
3. Error wrapping in Go

In Part 1 we talked about when to ignore error values in tests (answer: never), and also about how to write tests for the behaviour when errors are expected.

To do that, we need the function under test to return the appropriate error. Sometimes that’s easy, and sometimes it’s not. Let’s look at a couple of examples.

Simulating errors

If our function is supposed to return an error for invalid input, as in the format.Data example from Part 1 of this series, that’s fairly easy to test. We can just give it invalid input and see what it does.

In other cases it can be a little more difficult to arrange for the specified error to occur. For example, consider a function like this:

func ReadAll(r io.Reader) ([]byte, error) {
    data, err := io.ReadAll(r)
    if err != nil {
        return nil, err
    }
    return data, nil
}

(Listing reader/1)

If the function encounters a read error, it returns it. Now, we probably wouldn’t bother to test this behaviour in practice, because it’s so straightforward. But let’s try, just for the fun of it.

How could we arrange for a read error to happen in a test? That’s not so easy if we use something like a strings.Reader:

func TestReadAll_ReturnsAnyReadError(t *testing.T) {
    input := strings.NewReader("any old data")
    _, err := reader.ReadAll(input)
    if err == nil {
        t.Error("want error for broken reader, got nil")
    }
}

(Listing reader/1)

This will always fail, even when the function is correct, because reading from a strings.Reader never does produce an error.

How can we make the strings.Reader return an error when someone calls its Read method? We can’t. But we can implement our own very trivial io.Reader to do just that:

type errReader struct{}

func (errReader) Read([]byte) (int, error) {
    return 0, io.ErrUnexpectedEOF
}

(Listing reader/2)

This is manifestly useless as a reader, since it never reads anything, always just returning a fixed error instead. But that makes it just the thing to use in our test:

func TestReadAll_ReturnsAnyReadError(t *testing.T) {
    input := errReader{} // always returns error
    _, err := reader.ReadAll(input)
    if err == nil {
        t.Error("want error for broken reader, got nil")
    }
}

(Listing reader/2)

Because io.Reader is such a small interface, it’s very easy to implement it with whatever behaviour we need in a test. You could imagine more sophisticated kinds of fake reader, such as one that errors after a certain number of bytes have been read, or a ReadCloser that fails to close itself properly, and so on.

In fact, we don’t even need to implement this erroneous reader ourselves. It’s provided in the standard library iotest package as ErrReader, along with some other useful test readers such as TimeoutReader and HalfReader.

Testing that an error is not nil

There’s another kind of mistake it’s easy to make when testing error results, and it comes from a perfectly understandable thought process. “My function returns something and error,” says the programmer, “and tests are about comparing want and got. So I should compare both results against what I expect them to be.”

In other words, they’ll compare the “something” result with some expected value, which is fine. And they’ll also try to compare the “error” result with some expected value, which is where they run into a problem.

Let’s see an example. Remember our store.Open example from Part 1? It returns a *Store, if it can successfully open the store, or nil and some error if it can’t.

Suppose we’re writing a “want error” test for this behaviour. In other words, we’ll deliberately try to open some store that can’t be opened, and check that we get an error.

This is straightforward, because all we need to do is check that the err value is not nil. That’s all the “contract” promises, so that’s what we check:

func TestOpenGivesNonNilErrorForBogusFile(t *testing.T) {
    t.Parallel()
    _, err := store.Open("bogus file")
    if err == nil {
        t.Error("want error opening bogus store file")
    }
}

(Listing store/1)

As with the format.Data example, since we expect an error, we just ignore the other result value. We aren’t interested in it for this test.

Indeed, if we received it and assigned it to got, then the compiler would complain that we don’t then do anything with that variable:

got declared and not used

It would be easy to feel bullied by this into doing something with got, like comparing it against a want value. But we know that’s wrong.

What the compiler is really saying, in its gnomic way, is “Hey programmer, since you don’t seem to care about the value of got, you should ignore it using _.” Quite right, and that’s what we’ll do.

But supposing we (wrongly) think that we need to test that Open returns some specific error. How would we even do that? Well, we can try the want-and-got pattern:

func TestOpenGivesSpecificErrorForBogusFile(t *testing.T) {
    t.Parallel()
    want := errors.New("open bogus: no such file or directory")
    _, got := store.Open("bogus")
    if got != want {
        t.Errorf("wrong error: %v", got)
    }
}

(Listing store/1)

This seems plausible, I think you’ll agree. Yet, surprisingly, the test fails:

wrong error: open bogus: no such file or directory

Wait, what? That is the error we were expecting. So why are these two values not comparing equal?

Let’s make the problem even more explicit, by using errors.New to construct both values, and then comparing them:

func TestOneErrorValueEqualsAnother(t *testing.T) {
    t.Parallel()
    want := errors.New("Go home, Go, you're drunk")
    got := errors.New("Go home, Go, you're drunk")
    if got != want {
        t.Errorf("wrong error: %v", got)
    }
}

(Listing store/1)

Surely this can’t fail? On the contrary:

wrong error: Go home, Go, you're drunk

We might justifiably feel a little puzzled at this. We have two values constructed exactly the same way, and yet they don’t compare equal. Why not?

This is somewhat non-obvious behaviour, I admit, but Go is doing exactly what it’s supposed to. We can work out what’s happening, if we think it through.

What type does errors.New return? We know it implements the interface type error, but what concrete type is it? Let’s look at the source code of errors.New to see what it does:

func New(text string) error {
    return &errorString{text}
}

I bet you didn’t know the unexported struct type errorString existed, yet you’ve probably used it every day of your life as a Go programmer. But that’s not the important part.

The important part here is that errors.New returns a pointer. So when we construct error values by calling errors.New, the results we get are pointers.

Can you see now why they don’t compare equal? In fact, Go pointers never compare equal unless they point to the same object. That is, unless they’re literally the same memory address.

And that won’t be the case here, because they point to two distinct instances of the errorString struct that merely happen to contain the same message. This is why it doesn’t make sense to compare error values in Go using the != operator: they’ll never be equal.

What the programmer really wanted to know in this case was not “Do these two values point to the same piece of memory?”, but “Do these two values represent the same error?”

How can we answer that question?

String matching on errors is fragile

Here’s another attempt to answer the question of whether two values represent the same error:

func TestOpenGivesSpecificErrorStringForBogusFile(t *testing.T) {
    t.Parallel()
    want := errors.New("open bogus: no such file or directory")
    _, got := store.Open("bogus")
    if got.Error() != want.Error() {
        t.Errorf("wrong error: %v", got)
    }
}

(Listing store/1)

Instead of comparing the error values directly, we compare the strings produced by each error’s Error method.

But although this works, more or less, there’s something about it that doesn’t feel quite right. The whole point of error being an interface, after all, is that we shouldn’t have to care about what its string value is.

So if we construct our expected error using errors.New and some fixed string, and compare it with the error result from the function, that’s really just the same as comparing two strings.

That works great, right up until some well-meaning programmer makes a minor change to their error text, such as adding a comma. Bang! Broken tests all over the place. By doing this kind of comparison, we’ve made the test brittle.

There are no bad ideas, as they say, but let’s keep thinking.

Sentinel errors lose useful information

If string matching errors makes tests fragile, then maybe we could define some named error value for store.Open to return.

In other words, something like this:

var ErrUnopenable = errors.New("can't open store file")

This is called a sentinel error, and there are several examples in the standard library, such as io.EOF. It’s an exported identifier, so people using your package can compare your errors with that value.

This makes the test quite straightforward, because we can go back to our nice, simple scheme of comparing error values directly:

func TestOpenGivesErrUnopenableForBogusFile(t *testing.T) {
    t.Parallel()
    _, err := store.Open("bogus")
    if err != store.ErrUnopenable {
        t.Errorf("wrong error: %v", err)
    }
}

(Listing store/2)

This works, provided Open does indeed return this exact value in the error case. Well, we can arrange that:

func Open(path string) (*Store, error) {
    f, err := os.Open(path)
    if err != nil {
        return nil, ErrUnopenable // losing information about 'err'
    }
    // ...
}

(Listing store/2)

This looks okay, but it’s still not ideal. Actually, we lost some information here that might have been useful: specifically, why we couldn’t open the file. We had that information in the err variable, but then we threw it away and returned the fixed value ErrUnopenable instead.

So what do users eventually see? Just the value of ErrUnopenable, which is fixed:

can't open store file

This is very unhelpful. What store file? Why couldn’t it be opened? What was the error? What action could fix the problem? The user could be forgiven for feeling somewhat let down.

DR EVIL: Right, okay, people, you have to tell me these things, all right? I’ve been frozen for thirty years, okay? Throw me a frickin’ bone here! I’m the boss. Need the info.

—“Austin Powers: International Man of Mystery”

Actually, it would have been more helpful simply to return the err value directly, because it already contains everything the user needs to know:

open store.bin: permission denied

Much more informative! Unlike ErrUnopenable, this tells us not only which specific file couldn’t be opened, but also why not.

So a sentinel error value ErrUnopenable makes it possible to detect that kind of error programmatically, but at the expense of making the error message itself nearly useless. But did we even need to make this trade-off in the first place?

For example, do programs that use store.Open really need to distinguish “unopenable file” errors from other kinds of errors opening a store? Or is all they care about simply that there was some error?

Most of the time in Go programs, all we care about is that err is not nil. In other words, that there was some error. What it is specifically usually doesn’t matter, because the program isn’t going to take different actions for different errors. It’s probably just going to print the error message and then exit.

Getting back to the store.Open test, the user-facing behaviour that we care about is that, if the store can’t be opened, Open returns some error.

And that’s easy to detect, in a test or elsewhere. We can just compare it with nil, which is where we started:

func TestOpenGivesNonNilErrorForBogusFile(t *testing.T) {
    t.Parallel()
    _, err := store.Open("bogus file")
    if err == nil {
        t.Error("want error opening bogus store file")
    }
}

(Listing store/1)

In other words, if an error is intended to signal something to the user, then a sentinel error probably won’t be that helpful, because its value is fixed. We can’t use it to convey any dynamic information that might help the user solve the problem.

And if the system doesn’t need to know anything about the error except that it’s not nil, then we don’t need a sentinel error at all.

---

That’s it for Part 2. In Part 3, we’ll discuss error wrapping, distinguishing errors using errors.Is and errors.As, and implementing custom error types. Stay tuned!