Comparing Go error values
I believe you should never inspect the output of the
Error
method. TheError
method on theerror
interface exists for humans, not code. The contents of that string belong in a log file, or displayed on screen.
Comparing the string form of an error is, in my opinion, a code smell, and you should try to avoid it.
—Dave Cheney, “Don’t just check errors, handle them gracefully”
This is Part 2 of a series about errors and error testing in Go:
- Testing errors in Go
- Comparing Go error values
- Error wrapping in Go
In Part 1 we talked about when to ignore error values in tests (answer: never), and also about how to write tests for the behaviour when errors are expected.
To do that, we need the function under test to return the appropriate error. Sometimes that’s easy, and sometimes it’s not. Let’s look at a couple of examples.
Simulating errors
If our function is supposed to return an error for invalid input, as
in the format.Data
example from Part 1 of this series, that’s fairly
easy to test. We can just give it invalid input and see what it
does.
In other cases it can be a little more difficult to arrange for the specified error to occur. For example, consider a function like this:
func ReadAll(r io.Reader) ([]byte, error) {
, err := io.ReadAll(r)
dataif err != nil {
return nil, err
}
return data, nil
}
If the function encounters a read error, it returns it. Now, we probably wouldn’t bother to test this behaviour in practice, because it’s so straightforward. But let’s try, just for the fun of it.
How could we arrange for a read error to happen in a test? That’s not
so easy if we use something like a strings.Reader
:
func TestReadAll_ReturnsAnyReadError(t *testing.T) {
:= strings.NewReader("any old data")
input , err := reader.ReadAll(input)
_if err == nil {
.Error("want error for broken reader, got nil")
t}
}
This will always fail, even when the function is correct, because
reading from a strings.Reader
never does produce
an error.
How can we make the strings.Reader
return an error when
someone calls its Read
method? We can’t. But we can
implement our own very trivial io.Reader
to do
just that:
type errReader struct{}
func (errReader) Read([]byte) (int, error) {
return 0, io.ErrUnexpectedEOF
}
This is manifestly useless as a reader, since it never reads anything, always just returning a fixed error instead. But that makes it just the thing to use in our test:
func TestReadAll_ReturnsAnyReadError(t *testing.T) {
:= errReader{} // always returns error
input , err := reader.ReadAll(input)
_if err == nil {
.Error("want error for broken reader, got nil")
t}
}
Because io.Reader
is such a small interface, it’s very
easy to implement it with whatever behaviour we need in a test. You
could imagine more sophisticated kinds of fake reader, such as one that
errors after a certain number of bytes have been read, or a
ReadCloser
that fails to close itself properly, and so
on.
In fact, we don’t even need to implement this erroneous reader
ourselves. It’s provided in the standard library iotest
package
as ErrReader
,
along with some other useful test readers such as TimeoutReader
and HalfReader
.
Testing that an error is not
nil
There’s another kind of mistake it’s easy to make when testing error
results, and it comes from a perfectly understandable thought process.
“My function returns something and error,” says the programmer, “and
tests are about comparing want
and got
. So I
should compare both results against what I expect them to be.”
In other words, they’ll compare the “something” result with some expected value, which is fine. And they’ll also try to compare the “error” result with some expected value, which is where they run into a problem.
Let’s see an example. Remember our store.Open
example
from Part 1? It returns a
*Store
, if it can successfully open the store, or
nil
and some error if it can’t.
Suppose we’re writing a “want error” test for this behaviour. In other words, we’ll deliberately try to open some store that can’t be opened, and check that we get an error.
This is straightforward, because all we need to do is check that the
err
value is not nil
. That’s all the
“contract” promises, so that’s what we check:
func TestOpenGivesNonNilErrorForBogusFile(t *testing.T) {
.Parallel()
t, err := store.Open("bogus file")
_if err == nil {
.Error("want error opening bogus store file")
t}
}
As with the format.Data
example, since we
expect an error, we just ignore the other result value. We
aren’t interested in it for this test.
Indeed, if we received it and assigned it to got
, then
the compiler would complain that we don’t then do anything with
that variable:
got declared and not used
It would be easy to feel bullied by this into doing something with
got
, like comparing it against a want
value.
But we know that’s wrong.
What the compiler is really saying, in its gnomic way, is “Hey
programmer, since you don’t seem to care about the value of
got
, you should ignore it using _
.” Quite
right, and that’s what we’ll do.
But supposing we (wrongly) think that we need to test that
Open
returns some specific error. How would we
even do that? Well, we can try the want-and-got pattern:
func TestOpenGivesSpecificErrorForBogusFile(t *testing.T) {
.Parallel()
t:= errors.New("open bogus: no such file or directory")
want , got := store.Open("bogus")
_if got != want {
.Errorf("wrong error: %v", got)
t}
}
This seems plausible, I think you’ll agree. Yet, surprisingly, the test fails:
wrong error: open bogus: no such file or directory
Wait, what? That is the error we were expecting. So why are these two values not comparing equal?
Let’s make the problem even more explicit, by using
errors.New
to construct both values, and then
comparing them:
func TestOneErrorValueEqualsAnother(t *testing.T) {
.Parallel()
t:= errors.New("Go home, Go, you're drunk")
want := errors.New("Go home, Go, you're drunk")
got if got != want {
.Errorf("wrong error: %v", got)
t}
}
Surely this can’t fail? On the contrary:
wrong error: Go home, Go, you're drunk
We might justifiably feel a little puzzled at this. We have two values constructed exactly the same way, and yet they don’t compare equal. Why not?
This is somewhat non-obvious behaviour, I admit, but Go is doing exactly what it’s supposed to. We can work out what’s happening, if we think it through.
What type does errors.New
return? We know it implements
the interface type error
, but what concrete type
is it? Let’s look at the source code of errors.New
to see
what it does:
func New(text string) error {
return &errorString{text}
}
I bet you didn’t know the unexported struct type
errorString
existed, yet you’ve probably used it every day
of your life as a Go programmer. But that’s not the important part.
The important part here is that errors.New
returns a
pointer. So when we construct error values by calling
errors.New
, the results we get are pointers.
Can you see now why they don’t compare equal? In fact, Go pointers never compare equal unless they point to the same object. That is, unless they’re literally the same memory address.
And that won’t be the case here, because they point to two distinct
instances of the errorString
struct that merely happen to
contain the same message. This is why it doesn’t make sense to compare
error values in Go using the !=
operator: they’ll never be
equal.
What the programmer really wanted to know in this case was not “Do these two values point to the same piece of memory?”, but “Do these two values represent the same error?”
How can we answer that question?
String matching on errors is fragile
Here’s another attempt to answer the question of whether two values represent the same error:
func TestOpenGivesSpecificErrorStringForBogusFile(t *testing.T) {
.Parallel()
t:= errors.New("open bogus: no such file or directory")
want , got := store.Open("bogus")
_if got.Error() != want.Error() {
.Errorf("wrong error: %v", got)
t}
}
Instead of comparing the error values directly, we compare the
strings produced by each error’s Error
method.
But although this works, more or less, there’s something about it
that doesn’t feel quite right. The whole point of error
being an interface, after all, is that we shouldn’t have to
care about what its string value is.
So if we construct our expected error using
errors.New
and some fixed string, and compare it with the
error result from the function, that’s really just the same as comparing
two strings.
That works great, right up until some well-meaning programmer makes a minor change to their error text, such as adding a comma. Bang! Broken tests all over the place. By doing this kind of comparison, we’ve made the test brittle.
There are no bad ideas, as they say, but let’s keep thinking.
Sentinel errors lose useful information
If string matching errors makes tests fragile, then maybe we could
define some named error value for store.Open
to
return.
In other words, something like this:
var ErrUnopenable = errors.New("can't open store file")
This is called a sentinel error, and there are several
examples in the standard library, such as io.EOF
. It’s an
exported identifier, so people using your package can compare your
errors with that value.
This makes the test quite straightforward, because we can go back to our nice, simple scheme of comparing error values directly:
func TestOpenGivesErrUnopenableForBogusFile(t *testing.T) {
.Parallel()
t, err := store.Open("bogus")
_if err != store.ErrUnopenable {
.Errorf("wrong error: %v", err)
t}
}
This works, provided Open
does indeed return this exact
value in the error case. Well, we can arrange that:
func Open(path string) (*Store, error) {
, err := os.Open(path)
fif err != nil {
return nil, ErrUnopenable // losing information about 'err'
}
// ...
}
This looks okay, but it’s still not ideal. Actually, we lost
some information here that might have been useful: specifically,
why we couldn’t open the file. We had that information in the
err
variable, but then we threw it away and returned the
fixed value ErrUnopenable
instead.
So what do users eventually see? Just the value of
ErrUnopenable
, which is fixed:
can't open store file
This is very unhelpful. What store file? Why couldn’t it be opened? What was the error? What action could fix the problem? The user could be forgiven for feeling somewhat let down.
DR EVIL: Right, okay, people, you have to tell me these things, all right? I’ve been frozen for thirty years, okay? Throw me a frickin’ bone here! I’m the boss. Need the info.
Actually, it would have been more helpful simply to return the
err
value directly, because it already contains everything
the user needs to know:
open store.bin: permission denied
Much more informative! Unlike ErrUnopenable
, this tells
us not only which specific file couldn’t be opened, but also why
not.
So a sentinel error value ErrUnopenable
makes it
possible to detect that kind of error programmatically, but at
the expense of making the error message itself nearly useless. But did
we even need to make this trade-off in the first place?
For example, do programs that use store.Open
really
need to distinguish “unopenable file” errors from other kinds
of errors opening a store? Or is all they care about simply that there
was some error?
Most of the time in Go programs, all we care about is that
err
is not nil
. In other words, that there was
some error. What it is specifically usually doesn’t matter,
because the program isn’t going to take different actions for different
errors. It’s probably just going to print the error message and then
exit.
Getting back to the store.Open
test, the user-facing
behaviour that we care about is that, if the store can’t be opened,
Open
returns some error.
And that’s easy to detect, in a test or elsewhere. We can just
compare it with nil
, which is where we started:
func TestOpenGivesNonNilErrorForBogusFile(t *testing.T) {
.Parallel()
t, err := store.Open("bogus file")
_if err == nil {
.Error("want error opening bogus store file")
t}
}
In other words, if an error is intended to signal something to the user, then a sentinel error probably won’t be that helpful, because its value is fixed. We can’t use it to convey any dynamic information that might help the user solve the problem.
And if the system doesn’t need to know anything about the
error except that it’s not nil
, then we don’t need a
sentinel error at all.
That’s it for Part 2. In Part 3,
we’ll discuss error wrapping, distinguishing errors using
errors.Is
and errors.As
, and implementing
custom error types. Stay tuned!