Random testing in Go
How to write a million tests, and still finish work on time
I shot an arrow into the air,
It fell to earth, I knew not where.
—Henry Wadsworth Longfellow, “The Arrow and the Song”
This is the first of a four-part tutorial series on test fuzzing in Go:
- Random testing in Go
- Fuzz tests in Go
- Writing a Go fuzz target
- Finding bugs with fuzzing
Choosing good test cases for our Go programs can be a bit hit-and-miss. Sometimes we get lucky and find an input that causes incorrect behaviour, or even a crash, but in general, picking inputs at random isn’t a good way to find bugs.
Or is it? What if we leaned into that idea a little and used a lot of different inputs? Say, a million, or even a billion. With that many inputs, our chances of finding the one weird value that triggers a problem start to look pretty good.
Sounds like too much typing? I agree with you, so let’s get the computer to do the work instead. Isn’t that what they’re for, after all?
Generating random test inputs
Well, we know how to generate random numbers in Go: we can use math/rand (or, for a more cosmic level of randomness, bitfield/qrand). Let’s give it a try.
Suppose we have a pair of functions Encode and Decode. It doesn’t matter what they do exactly, but we can assume that if you Encode something, then Decode the result, you should get back what you started with… if the functions are working correctly, that is.
Here’s a test that generates a random integer between 0 and 9 and sends it on a round trip through two functions, Encode and Decode:
import "math/rand"
func TestEncodeFollowedByDecodeGivesStartingValue(t *testing.T) {
t.Parallel()
input := rand.Intn(10)
encoded := codec.Encode(input)
t.Logf("encoded value: %#v", encoded)
want := input
got := codec.Decode(encoded)
// after the round trip, we should get what we started with
if want != got {
t.Errorf("want %d, got %d", want, got)
}
}You might worry that, if the value is truly random, then we could end up with a flaky test. If there’s a bug in Encode or Decode that’s only triggered by certain inputs, then won’t a test like this sometimes pass and sometimes fail?
That’s definitely a possibility. One way to avoid it is to seed the random number generator with some fixed value: that way, the generated sequence will always be the same, making our tests deterministic.
For example, we could write something like this, which will create a new random generator just for the tests, seeded with the value 1:
var rng = rand.New(rand.NewSource(1))We don’t have to use 1 as the seed; any fixed value would do. The point is that, given a certain seed, calling rng.Intn will always produce the same sequence of values:
fmt.Println(rng.Intn(10)) // 1
fmt.Println(rng.Intn(10)) // 7
fmt.Println(rng.Intn(10)) // 7
fmt.Println(rng.Intn(10)) // 9
fmt.Println(rng.Intn(10)) // 1You might wonder how it is that the “random” generator always produces the same sequence from the same seed: isn’t that the opposite of random? That’s a very interesting question that we don’t have the space to go into properly here, but you’ll be glad to know that it’s thoroughly explored in my book “Explore Go: Cryptography”.
Randomly permuting a set of known inputs
A nice way to use randomness without causing flaky tests, or having to manually seed the random generator, is to permute a set of inputs—that is, rearrange them in some random order.
For example, the following code generates a slice containing the integers from 0 to 99, ordered randomly:
inputs := rand.Perm(100)
for _, n := range inputs {
... // test with input n
}The sequence of 100 integers generated by rand.Perm(100) is not random itself, since each value from 0 to 99 will be represented exactly once. That wouldn’t be true of randomly chosen numbers, where some values would occur many times and others not at all.
Instead, this sequence is randomly permuted (that is, randomly ordered). It’s like shuffling a deck of cards: you know that each card will still show up exactly once, you just don’t know when.
Just like rand.Intn, the result will be different for every test run, unless you create your own random generator initialised with a known seed.
Property-based testing
Randomness can be a good way of finding interesting new test cases that you might not come up with by yourself. For example, if your function takes integers, you might have tried testing it with 1, 5, 7, and so on. You might not have thought of trying zero as an input, but the random generator would likely produce that at some point. And if your function breaks when given zero, that’s something you’d certainly want to know.
Good testers, in general, are good at suggesting inputs that break the program because the programmer didn’t think of them. Randomness can help with that process.
One problem with randomised test inputs that may already have occurred to you is this: if we don’t know in advance what input we’re going to use, we can’t know what answer to expect as the result.
For example:
func TestSquareGivesCorrectResult(t *testing.T) {
t.Parallel()
input := rand.Intn(100)
got := square.Square(input)
want := ... // uh-oh, what to put here?What’s our want value here? We don’t know, because we don’t know what the value of input will be when the test runs, and thus what its square would be. If we knew that input would be 10, for example, then we could have the test expect the answer 100, but we don’t know that. We’re stuck.
And we don’t want to try to compute the expected result in the test, because clearly we could get that computation wrong. In the most pernicious case, we might end up using the same code path to compute it in the test as we do in the system, so we’d end up testing nothing at all.
If we can’t predict what the exact result of Square will be for a given input, is there anything we can say about it in general terms?
Actually, there is something we can say: it shouldn’t be negative! No matter what the input, if Square returns a negative result, something’s gone wrong. So although we can’t predict the exact result if the system is correct, we can still identify some properties it should have.
So we could write a test that calls Square with lots of different inputs, and checks that the result is never negative:
func TestSquareResultIsAlwaysNonNegative(t *testing.T) {
t.Parallel()
inputs := rand.Perm(100)
for _, n := range inputs {
t.Run(strconv.Itoa(n), func(t *testing.T) {
got := square.Square(n)
if got < 0 {
t.Errorf("Square(%d) is negative: %d", n, got)
}
})
}
}This approach is sometimes called property-based testing, to distinguish it from what we’ve been doing up to now, which we might call example-based testing.
Another way to think about it is that property-based tests describe the behaviour of the system, not in terms of exact values, but in terms of invariants: things that don’t change about the result, no matter what the input is. In the case of Square, for example, its result should invariably be positive.
Randomised, property-based testing helps to fix the problem that maybe the function only works for the specific examples we thought of. And although we generate the inputs randomly, once we find some value that triggers a bug, it should then simply become part of our conventional example-based tests.
We could do this by manually adding such values to the set of inputs in a table test, for example, but there’s no need. Go provides an automated way to turn randomly-generated breaking inputs into static test cases for us, using what’s called fuzz testing.
In Part 2 of this series, we’ll introduce Go fuzz tests and see how they can be used to find rare inputs that trigger bugs. Stay tuned!
