Racing with disaster: data races in Go
BERNARD: I’m going to have to let you go.
MANNY: What? But I sold a lot of books. I got on well with all the customers.
BERNARD: It’s not that kind of operation.
—Black Books
Imagine you’re browsing in a bookstore: one of those delightful pokey old ones with endless shelves, rooms leading to other rooms, and books piled high on every available surface. Somewhere in the labyrinth, a well-fed ginger cat is sleeping precariously on a stack of Agatha Christies.
You spot something that looks interesting: a copy of Kip Thorne’s Black Holes and Time Warps. As you bend down to look closer, though, the title shimmers and changes. It’s now David Deutsch’s The Fabric of Reality. It’s priced at $10, but as you watch, the label distorts and the numbers flow before your eyes. It just went up to $15.
Well, that’s weird! You notice that, where there was only one copy before, there are now twelve huddled together on the shelf. You look round to see if anyone else is noticing this odd behaviour, and when you look back, you find the books are all gone. The one you were holding has morphed into Douglas Hofstadter’s Gödel, Escher, Bach.
We’d be puzzled indeed if this sort of thing happened even in the otherworldly surroundings of a vintage bookstore. But it’s the kind of problem that we run into all the time in concurrent computer systems, where seemingly stable things can change, fluctuate, or disappear when you’re not looking at them, and even—disconcertingly—when you are. It’s called a data race.
In Starving, sleeping, and yielding, we wrote a simple concurrent program in Go that prints interleaved “hello” messages from two different goroutines:
package main
import (
"fmt"
"time"
)
func goroutineB() {
for i := range 10 {
fmt.Println("Hello from goroutine B!", i)
time.Sleep(10 * time.Millisecond)
}
}
func main() {
go goroutineB()
for i := range 10 {
fmt.Println("Hello from goroutine A!", i)
time.Sleep(10 * time.Millisecond)
}
}And we saw that because of the non-deterministic behaviour of the scheduler (as explored in Go, go, goroutines), we can’t know in principle what the exact output of a program like this will be. We can predict that we’ll see messages from both goroutines, but we don’t know in exactly what order they’ll appear: this uncertainty is characteristic of concurrent programs.
It’s not really a problem with this program, though, because the goroutines are totally independent: they don’t share any data. Unfortunately, being able to share data between goroutines is precisely what makes Go’s concurrency so useful.
A shared variable
For example, let’s modify this program so that each goroutine prints
the value of a shared package-level variable message:
var message = "Hello"
func goroutineB() {
for range 10 {
fmt.Println(message, "from goroutine B!")
time.Sleep(10 * time.Millisecond)
}
}
func main() {
go goroutineB()
for range 10 {
fmt.Println(message, "from goroutine A!")
time.Sleep(10 * time.Millisecond)
}
}Here’s the output:
Hello from goroutine A!
Hello from goroutine B!
Hello from goroutine B!
...
Hello from goroutine A!Racing goroutines
Does a data race exist in this program? We can decide that by asking two questions. First, are multiple goroutines accessing the same variable? Second, is at least one of those accesses a write?
If the answer to both questions is yes, then we have a data race. It
turns out that we’re safe for now, because even though the two
goroutines are sharing message, all their accesses are
reads; no writes.
Let’s live a little dangerously, then. If we have one of our
goroutines modify the message variable, then we
should have a data race, according to the rules. We’ll change goroutine
B as follows:
func goroutineB() {
message = "Goodbye"
for range 10 {
fmt.Println(message, "from goroutine B!")
time.Sleep(10 * time.Millisecond)
}
}So, when goroutine B runs for the first time, it will set
message to the string "Goodbye". Let’s see
what the output looks like:
Hello from goroutine A!
Goodbye from goroutine B!
Goodbye from goroutine B!
Goodbye from goroutine A!
...
Goodbye from goroutine B!Fair enough. For a given iteration of goroutine A, then, will it print “Hello” or “Goodbye”? The answer is that we can’t know in advance, and that’s the significance of a data race. The output of goroutine A depends in this case on whether or not goroutine B has ever executed, and that in turn depends on what goroutine A does.
Data races in concurrent systems
What are the consequences of this for a more realistic concurrent program? Suppose that our sleepy bookstore, Happy Fun Books, has two point-of-sale terminals, both accessing the same book catalog concurrently. And let’s suppose that two customers are both trying to buy the last copy of Black Holes and Time Warps.
We haven’t written a program to do this yet, but let’s imagine
there’s a command called buy that, among other things,
checks the stock level of the given book and decreases it by 1. If you
try to buy abc when there are zero copies of book
abc in stock, clearly that can’t succeed, so you get an
error message saying “sorry, out of stock”.
So, if there’s initially one copy of abc in the catalog,
then what happens when customers A and B both concurrently try to buy
it? In theory, one of them must win the race: if customer A is a
microsecond ahead of customer B in placing their order, then they’ll get
the last copy, and the second attempt to buy abc will fail
because there are now zero copies left.
But in practice, computer systems store temporary copies of data in cache memory. We can imagine that our bookstore terminals might use a similar kind of cache arrangement, and in that case these multiple copies of the book data can get out of sync, with alarming consequences.
In my book The Deeper Love of Go, we’ll
actually write the buy program, and everything else
required to run a bookstore, using a networked database system in Go.
It’s not only an introduction to programming for complete beginners, but
also a primer on concurrency, distributed systems, and locks for those
who missed out on (or snoozed through) the relevant CompSci classes. If
you’re enjoying this extract, check out the book!
Double access problems
Here’s what could happen with our duelling customers:
Customer A runs
buy abc, setting the number of copies ofabcin its local cache of the catalog to zero. But customer B also has a cached version of the catalog that now disagrees with customer A’s version, because it still says there is one copy ofabc, instead of zero.Concurrently, customer B runs
buy abc. This succeeds because, as far as B’s cache is concerned, there’s still one copy ofabcleft in the catalog.Both customer A and customer B’s caches now agree again. Specifically, they agree that there should be zero copies of
abcin the catalog, so there’s no conflict, and the system doesn’t detect any problem.But there is a problem! Only one of the two customers can walk out of the store with the book. Yet they’ve both bought and paid for it, and the bookstore system failed to prevent the same copy from being sold twice.
What can we do about this?
Detecting races
Let’s write a Go program that demonstrates a simple version of the duelling-customers problem, and then we’ll see if we can detect and fix it.
Clash of the customers
We’ll represent each customer by a goroutine, and each goroutine will
attempt to “buy” the same book by decrementing a shared
copies variable:
func main() {
copies := 1
go func() {
if copies > 0 {
copies--
fmt.Println("Customer A got the book")
}
}()
go func() {
if copies > 0 {
copies--
fmt.Println("Customer B got the book")
}
}()
time.Sleep(time.Second)
}Here, we’re setting up a variable copies, representing
the number of copies of the book in stock. It starts at 1.
Next, we create two goroutines, representing customers A and B, each
of which does the same thing: it checks to see if copies is
greater than zero, and if so, it “buys” the book by setting
copies to zero and printing a message saying which customer
got it.
In total, then, we have three goroutines now, including the
main goroutine, which does nothing but call time.Sleep, to
give the two “customer” goroutines a chance to run, and to make sure the
program doesn’t end too soon.
Demonstrating the race
When this program runs, then, the two customer goroutines should both
get a chance to run, and they will race each other for access to
copies. Sometimes customer A will get there first, and at
other times, customer B: the result should be more or less random.
Let’s see what happens when we run this program ten times in a row, then:
Customer B got the book
Customer A got the book
Customer A got the book
Customer B got the book
Customer A got the book
Customer B got the book
Customer B got the book
Customer B got the book
Customer A got the book
Customer A got the bookWe already knew that a data race is present here in theory, because there are multiple goroutines accessing the same variable, and at least one of those accesses is a write. But now we’ve demonstrated the race in practice: we can’t tell in advance whether customer A or B will get the book. The outcome of the program is non-deterministic, in other words.
And we know that in a more realistic simulation of a multi-user system that includes cacheing, a data race means that we can get undesirable results such as both customers apparently purchasing the same copy.
Go has a useful tool to help us here, called the race
detector. If we use the -race flag with the
go command, the Go runtime will use the race detector to
find and report any data races in the program.
We feel it should report one here, so let’s find out:
go run -race main.go
Customer B got the book
==================
WARNING: DATA RACE
Read at 0x00c000090038 by goroutine 6:
main.main.func1()
.../love/copies/1/main.go:11 +0x2c
Previous write at 0x00c000090038 by goroutine 7:
main.main.func2()
.../love/copies/1/main.go:18 +0x50
Goroutine 6 (running) created at:
main.main()
.../love/copies/1/main.go:10 +0xa0
Goroutine 7 (running) created at:
main.main()
.../love/copies/1/main.go:16 +0x104
==================
Found 1 data race(s)
exit status 66The race detector output
We know from the definition of a data race that there must be at least one write to the variable and at least one other conflicting access, and indeed that’s the case. Here’s where the write occurs:
Previous write... by goroutine 7:
main.main.func2()
.../love/copies/1/main.go:18That’s referring to this line in the customer B goroutine (“goroutine 7”, according to the arbitrary numbering scheme used by the runtime):
copies--Customer B is trying to decrement copies, thus updating
the value of the variable, but this is conflicting with a concurrent
read by the customer A goroutine (“goroutine 6”):
Read... by goroutine 6:
main.main.func1()
.../love/copies/1/main.go:11That refers to this line in the customer A goroutine:
if copies > 0 {So, customer A is trying to read the variable, while customer B is
concurrently trying to write to it. Go has successfully detected the
data race in this program. It’s a good idea to always use the
-race flag when you test programs with multiple goroutines,
just to make sure you haven’t missed any potential races.
If the race detector is so super-terrific, you might ask, then why isn’t it just turned on all the time? The answer is that it works by adding instrumentation to every piece of data in your program to check if it’s being accessed concurrently. Those checks slow down the program a lot, so we wouldn’t want them enabled when running the program for real, but they remain very useful for testing.
Having proved a data race exists, though, how can we fix it? In the next post, we’ll design a scheme for locking the shared data to prevent conflicting reads and writes, and we’ll see how to implement it in Go. Don’t miss it!
