Unit Testing

to framework, or not to framework?

28 April 2014

Bryan Mills

disclaimer

The opinions and advice presented here are my own, and are not necessarily shared by my employer or the Go Authors.

(…but I hope they would generally agree!)

background: Go testing

If you haven't already seen it, have a look at Andrew Gerrand's Testing Techniques talk from Google I/O.

It covers:

background: frameworks

In addition to the tools in the Go standard library, there are a number of "testing frameworks" available for Go, including:

These frameworks advocate or enable different styles of testing.

objective

In this talk, I will contrast a test using the Ginkgo framework with an equivalent test using the testing package directly.

I believe that the comparison clearly illustrates that the additional framework provides little to no advantage, and in fact harms the overall usefulness of the tests.

Sadly, I did not have time to cover:

a Ginkgo example

set up the framework

func TestSynths(t *testing.T) {
    RegisterFailHandler(Fail)
    RunSpecs(t, "Synths Suite")
}

describe the test environment

var _ = Describe("Synth", func() {
    const (
        period = 1024
    )

    var (
        amplitude stream.Sample
        synth     Synth
        st        stream.Stream
        err       error
        buf       []stream.Sample
    )

    BeforeEach(func() {
        buf = make([]stream.Sample, period)
    })

construct the thing to be tested

    Describe("signal properties", func() {
        JustBeforeEach(func() {
            st, err = synth.Play(0)
            Expect(err).NotTo(HaveOccurred())

            st.Fill(buf)
        })

If Play returns an error, how do we know what synth it was using?

(Guess I'd better hope it adds enough context…)

test a property

        AfterEach(func() {
            It("should hit peak amplitude", func() {
                var (
                    max = stream.Sample(-stream.MaxSample)
                    min = stream.Sample(stream.MaxSample)
                )
                for _, s := range buf {
                    if s > max {
                        max = s
                    }
                    if s < min {
                        min = s
                    }
                }
                Expect(max).To(Equal(amplitude))
                Expect(min).To(Equal(-amplitude))
            })

Ok, now this is starting to look like a test. Not bad.

test another property

            It("should average zero", func() {
                sum := big.NewRat(0, 1)
                inc := new(big.Rat)
                for _, s := range buf {
                    sum.Add(sum, inc.SetInt64(int64(s)))
                }
                mean := new(big.Rat).Quo(sum, big.NewRat(period, 1))
                Expect(mean).To(Equal(big.NewRat(0, 1)))
            })

add some conditions to test in

        Context("with a Sine synth", func() {
            JustBeforeEach(func() {
                synth = NewSine(big.NewRat(period, 1), amplitude)
            })

            Context("with high amplitude", func() {
                BeforeEach(func() {
                    amplitude = stream.MaxSample
                })
            })

            Context("with low amplitude", func() {
                BeforeEach(func() {
                    amplitude = 10
                })
            })
        })

ok, time to run the tests!

package main

import (
	"math/big"
	"os"
	"os/exec"
	"syscall"
	"testing"

	"bitbucket.org/bcmills/harmonolog/stream"
	. "bitbucket.org/bcmills/harmonolog/synths"

	. "github.com/onsi/ginkgo"
	. "github.com/onsi/gomega"
)

func main() {
	cmd := exec.Command("go", "test", "bitbucket.org/bcmills/gsp/2014-08-28/ginkgo_1_synths_test.go", "--ginkgo.noColor")
	cmd.Stdout = os.Stdout
	cmd.Stdin = os.Stdin
	cmd.Run()
	os.Exit(cmd.ProcessState.Sys().(syscall.WaitStatus).ExitStatus())
}

func TestSynths(t *testing.T) {
    RegisterFailHandler(Fail)
    RunSpecs(t, "Synths Suite")
}

// START-SLIDE1 OMIT
var _ = Describe("Synth", func() {
	const (
		period = 1024
	)

	var (
		amplitude stream.Sample
		synth     Synth
		st        stream.Stream
		err       error
		buf       []stream.Sample
	)

	BeforeEach(func() {
		buf = make([]stream.Sample, period)
	})
	// END-SLIDE1 OMIT

	// START-SLIDE2 OMIT
	Describe("signal properties", func() {
		JustBeforeEach(func() {
			st, err = synth.Play(0)
			Expect(err).NotTo(HaveOccurred())

			st.Fill(buf)
		})
		// END-SLIDE2 OMIT

		// START-SLIDE3 OMIT
		AfterEach(func() {
			It("should hit peak amplitude", func() {
				var (
					max = stream.Sample(-stream.MaxSample)
					min = stream.Sample(stream.MaxSample)
				)
				for _, s := range buf {
					if s > max {
						max = s
					}
					if s < min {
						min = s
					}
				}
				Expect(max).To(Equal(amplitude))
				Expect(min).To(Equal(-amplitude))
			})
			// END-SLIDE3 OMIT

			// START-SLIDE4 OMIT
			It("should average zero", func() {
				sum := big.NewRat(0, 1)
				inc := new(big.Rat)
				for _, s := range buf {
					sum.Add(sum, inc.SetInt64(int64(s)))
				}
				mean := new(big.Rat).Quo(sum, big.NewRat(period, 1))
				Expect(mean).To(Equal(big.NewRat(0, 1)))
			})
			// END-SLIDE4 OMIT
		})

		// START-SLIDE5 OMIT
		Context("with a Sine synth", func() {
			JustBeforeEach(func() {
				synth = NewSine(big.NewRat(period, 1), amplitude)
			})

			Context("with high amplitude", func() {
				BeforeEach(func() {
					amplitude = stream.MaxSample
				})
			})

			Context("with low amplitude", func() {
				BeforeEach(func() {
					amplitude = 10
				})
			})
		})
		// END-SLIDE5 OMIT
	})
})

yay, they passed!

wait, did they even run?

oops.

ok, let's fix it.

        AssertPeakAmplitude := func() {
            It("should hit peak amplitude", func() {
                var (
                    max = stream.Sample(-stream.MaxSample)
                    min = stream.Sample(stream.MaxSample)
                )
                for _, s := range buf {
                    if s > max {
                        max = s
                    }
                    if s < min {
                        min = s
                    }
                }
                Expect(max).To(Equal(amplitude))
                Expect(min).To(Equal(-amplitude))
            })
        }

argh, boilerplate!

            Context("with high amplitude", func() {
                BeforeEach(func() {
                    amplitude = stream.MaxSample
                })

                AssertPeakAmplitude()
                AssertMeanZero()
            })

            Context("with low amplitude", func() {
                BeforeEach(func() {
                    amplitude = 10
                })

                AssertPeakAmplitude()
                AssertMeanZero()
            })

ok, time to run!

package main

import (
	"math/big"
	"os"
	"os/exec"
	"syscall"
	"testing"

	"bitbucket.org/bcmills/harmonolog/stream"
	. "bitbucket.org/bcmills/harmonolog/synths"

	. "github.com/onsi/ginkgo"
	. "github.com/onsi/gomega"
)

func main() {
	cmd := exec.Command("go", "test", "bitbucket.org/bcmills/gsp/2014-08-28/ginkgo_2_synths_test.go", "--ginkgo.noColor")
	cmd.Stdout = os.Stdout
	cmd.Stdin = os.Stdin
	cmd.Run()
	os.Exit(cmd.ProcessState.Sys().(syscall.WaitStatus).ExitStatus())
}

func TestSynths(t *testing.T) {
    RegisterFailHandler(Fail)
    RunSpecs(t, "Synths Suite")
}

var _ = Describe("Synth", func() {
	const (
		period = 1024
	)

	// START-PARAMS OMIT
	var (
		amplitude stream.Sample
		synth     Synth
		st        stream.Stream
		err       error
		buf       []stream.Sample
	)

	BeforeEach(func() {
		buf = make([]stream.Sample, period)
	})
	// END-PARAMS OMIT

	Describe("signal properties", func() {
		JustBeforeEach(func() {
			st, err = synth.Play(0)
			Expect(err).NotTo(HaveOccurred())

			st.Fill(buf)
		})

		// START-ASSERTS OMIT
		AssertPeakAmplitude := func() {
			It("should hit peak amplitude", func() {
				var (
					max = stream.Sample(-stream.MaxSample)
					min = stream.Sample(stream.MaxSample)
				)
				for _, s := range buf {
					if s > max {
						max = s
					}
					if s < min {
						min = s
					}
				}
				Expect(max).To(Equal(amplitude))
				Expect(min).To(Equal(-amplitude))
			})
		}
		// END-ASSERTS OMIT

		AssertMeanZero := func() {
			It("should average zero", func() {
				sum := big.NewRat(0, 1)
				inc := new(big.Rat)
				for _, s := range buf {
					sum.Add(sum, inc.SetInt64(int64(s)))
				}
				mean := new(big.Rat).Quo(sum, big.NewRat(period, 1))
				Expect(mean).To(Equal(big.NewRat(0, 1)))
			})
		}

		Context("with a Sine synth", func() {
			JustBeforeEach(func() {
				synth = NewSine(big.NewRat(period, 1), amplitude)
			})

			// START-ASSERT-CONTEXT OMIT
			Context("with high amplitude", func() {
				BeforeEach(func() {
					amplitude = stream.MaxSample
				})

				AssertPeakAmplitude()
				AssertMeanZero()
			})

			Context("with low amplitude", func() {
				BeforeEach(func() {
					amplitude = 10
				})

				AssertPeakAmplitude()
				AssertMeanZero()
			})
			// END-ASSERT-CONTEXT OMIT
		})
	})
})

huh?!

Test Panicked
runtime error: invalid memory address or nil pointer dereference
/usr/local/go/src/pkg/runtime/panic.c:552

Full Stack Trace
/home/bryan/src/bitbucket.org/bcmills/gsp/2014-08-28/ginkgo_2_synths_test.go:54 (0x44602e)
func.003: st, err = synth.Play(0)
/home/bryan/src/bitbucket.org/bcmills/gsp/2014-08-28/ginkgo_2_synths_test.go:28 (0x445df4)
TestSynths: RunSpecs(t, "Synths Suite")

What's the problem here?

I don't really know, and the test framework isn't exactly helping me out. Maybe JustBeforeEach runs in outside-to-inside order, unlike BeforeEach?

inside-out control flow

    var (
        amplitude stream.Sample
        synth     Synth
        st        stream.Stream
        err       error
        buf       []stream.Sample
    )

    BeforeEach(func() {
        buf = make([]stream.Sample, period)
    })

inside-out control flow

fix it with boilerplate.

package main

import (
	"math/big"
	"os"
	"os/exec"
	"syscall"
	"testing"

	"bitbucket.org/bcmills/harmonolog/stream"
	. "bitbucket.org/bcmills/harmonolog/synths"

	. "github.com/onsi/ginkgo"
	. "github.com/onsi/gomega"
)

func main() {
	cmd := exec.Command("go", "test", "bitbucket.org/bcmills/gsp/2014-08-28/ginkgo_3_synths_test.go", "--ginkgo.noColor")
	cmd.Stdout = os.Stdout
	cmd.Stdin = os.Stdin
	cmd.Run()
	os.Exit(cmd.ProcessState.Sys().(syscall.WaitStatus).ExitStatus())
}

func TestSynths(t *testing.T) {
	RegisterFailHandler(Fail)
	RunSpecs(t, "Synths Suite")
}

var _ = Describe("Synth", func() {
	const (
		period = 1024
	)

	var (
		amplitude stream.Sample
		synth     Synth
		st        stream.Stream
		err       error
		buf       []stream.Sample
	)

	BeforeEach(func() {
		buf = make([]stream.Sample, period)
	})

	Describe("signal properties", func() {
		JustBeforeEach(func() {
			st, err = synth.Play(0)
			Expect(err).NotTo(HaveOccurred())

			st.Fill(buf)
		})

		AssertPeakAmplitude := func() {
			It("should hit peak amplitude", func() {
				var (
					max = stream.Sample(-stream.MaxSample)
					min = stream.Sample(stream.MaxSample)
				)
				for _, s := range buf {
					if s > max {
						max = s
					}
					if s < min {
						min = s
					}
				}
				Expect(max).To(Equal(amplitude))
				Expect(min).To(Equal(-amplitude))
			})
		}

		AssertMeanZero := func() {
			It("should average zero", func() {
				sum := big.NewRat(0, 1)
				inc := new(big.Rat)
				for _, s := range buf {
					sum.Add(sum, inc.SetInt64(int64(s)))
				}
				mean := new(big.Rat).Quo(sum, big.NewRat(period, 1))
				Expect(mean).To(Equal(big.NewRat(0, 1)))
			})
		}

		Context("with a Sine synth", func() {
            Context("with high amplitude", func() {
                BeforeEach(func() {
                    amplitude = stream.MaxSample
                    synth = NewSine(big.NewRat(period, 1), amplitude)
                })

                AssertPeakAmplitude()
                AssertMeanZero()
            })

            Context("with low amplitude", func() {
                BeforeEach(func() {
                    amplitude = 10
                    synth = NewSine(big.NewRat(period, 1), amplitude)
                })

                AssertPeakAmplitude()
                AssertMeanZero()
            })
		})
	})
})

let's make it fail for real

package main

import (
	"math/big"
	"os"
	"os/exec"
	"syscall"
	"testing"

	"bitbucket.org/bcmills/harmonolog/stream"
	. "bitbucket.org/bcmills/harmonolog/synths"

	. "github.com/onsi/ginkgo"
	. "github.com/onsi/gomega"
)

func main() {
	cmd := exec.Command("go", "test", "bitbucket.org/bcmills/gsp/2014-08-28/ginkgo_4_synths_test.go", "--ginkgo.noColor")
	cmd.Stdout = os.Stdout
	cmd.Stdin = os.Stdin
	cmd.Run()
	os.Exit(cmd.ProcessState.Sys().(syscall.WaitStatus).ExitStatus())
}

func TestSynths(t *testing.T) {
	RegisterFailHandler(Fail)
	RunSpecs(t, "Synths Suite")
}

var _ = Describe("Synth", func() {
	const (
		period = 1024
	)

	var (
		amplitude stream.Sample
		synth     Synth
		st        stream.Stream
		err       error
		buf       []stream.Sample
	)

	BeforeEach(func() {
		buf = make([]stream.Sample, period)
	})

	Describe("signal properties", func() {
		JustBeforeEach(func() {
			st, err = synth.Play(0)
			Expect(err).NotTo(HaveOccurred())

			st.Fill(buf)
		})

		AssertPeakAmplitude := func() {
			It("should hit peak amplitude", func() {
				var (
					max = stream.Sample(-stream.MaxSample)
					min = stream.Sample(stream.MaxSample)
				)
				for _, s := range buf {
					if s > max {
						max = s
					}
					if s < min {
						min = s
					}
				}
				Expect(max).To(Equal(amplitude))
				Expect(min).To(Equal(-amplitude))
			})
		}

        AssertMeanZero := func() {
            It("should average zero", func() {
                sum := big.NewRat(0, 1)
                inc := new(big.Rat)
                for _, s := range buf {
                    sum.Add(sum, inc.SetInt64(int64(s)))
                }
                mean := new(big.Rat).Quo(sum, big.NewRat(period, 1))
                Expect(mean).To(Equal(big.NewRat(42, 1)))
            })
        }

		Context("with a Sine synth", func() {
			Context("with high amplitude", func() {
				BeforeEach(func() {
					amplitude = stream.MaxSample
					synth = NewSine(big.NewRat(period, 1), amplitude)
				})

				AssertPeakAmplitude()
				AssertMeanZero()
			})

			Context("with low amplitude", func() {
				BeforeEach(func() {
					amplitude = 10
					synth = NewSine(big.NewRat(period, 1), amplitude)
				})

				AssertPeakAmplitude()
				AssertMeanZero()
			})
		})
	})
})

bad context

[Fail] Synth signal properties with a Sine synth with low amplitude [It] should average zero

bad detail

Expected
    <*big.Rat | 0xc20806c100>: {
        a: {neg: false, abs: nil},
        b: {neg: false, abs: []},
    }
to equal
    <*big.Rat | 0xc20806cac0>: {
        a: {neg: false, abs: [0x2a]},
        b: {neg: false, abs: []},
    }

Gomega can't know whether the String method is useful, so it avoids it - and we're left to parse the failures from the human-entered text.

We could fix it with a custom matcher… but that's more boilerplate.

ambiguous conditions

What does gomega.Equal really do for a big.Rat?

I can't easily see in the test code. Have to look it up.

Oh, it's using reflect.DeepEqual. Why didn't we just say so in the first place?

(Gomega tests aren't normal Go programs - Gomega is more like another language embedded within Go.)

Test code is code.

Tests are programs written to determine whether an API satisfies certain properties.

If the code under test is written in Go, why shouldn't the rest of the test be?

a Go example

write some Go functions

func minSample(buf []stream.Sample) stream.Sample {
    min := stream.Sample(stream.MaxSample)
    for _, s := range buf {
        if s < min {
            min = s
        }
    }
    return min
}

func mean(buf []stream.Sample) *big.Rat {
    sum := big.NewRat(0, 1)
    inc := new(big.Rat)
    for _, s := range buf {
        sum.Add(sum, inc.SetInt64(int64(s)))
    }
    return new(big.Rat).Quo(sum, big.NewRat(int64(len(buf)), 1))
}

describe the test environment

func TestSine(t *testing.T) {
    const period = 1024
    amplitudes := []stream.Sample{stream.MaxSample, 10}

    for _, amplitude := range amplitudes {

(Note that this is a very simple form of table-driven test.)

construct the thing to be tested

        synth := NewSine(big.NewRat(period, 1), amplitude)
        st, err := synth.Play(0)
        if err != nil {
            t.Errorf("NewSine(%d, %d).Play(0) = _, %v", period, amplitude, err)
            continue
        }

        buf := make([]stream.Sample, period)
        st.Fill(buf)

This is almost exactly what real uses of the synth package look like!

test the thing with the functions!

package main

import (
	"math/big"
	"os"
	"os/exec"
	"syscall"
	"testing"

	"bitbucket.org/bcmills/harmonolog/stream"
	. "bitbucket.org/bcmills/harmonolog/synths"
)

func main() {
	cmd := exec.Command("go", "test", "bitbucket.org/bcmills/gsp/2014-08-28/synths_test.go")
	cmd.Stdout = os.Stdout
	cmd.Stdin = os.Stdin
	cmd.Run()
	os.Exit(cmd.ProcessState.Sys().(syscall.WaitStatus).ExitStatus())
}

func maxSample(buf []stream.Sample) stream.Sample {
	max := stream.Sample(-stream.MaxSample)
	for _, s := range buf {
		if s > max {
			max = s
		}
	}
	return max
}

// START-HELPERS OMIT
func minSample(buf []stream.Sample) stream.Sample {
	min := stream.Sample(stream.MaxSample)
	for _, s := range buf {
		if s < min {
			min = s
		}
	}
	return min
}

func mean(buf []stream.Sample) *big.Rat {
	sum := big.NewRat(0, 1)
	inc := new(big.Rat)
	for _, s := range buf {
		sum.Add(sum, inc.SetInt64(int64(s)))
	}
	return new(big.Rat).Quo(sum, big.NewRat(int64(len(buf)), 1))
}

// END-HELPERS OMIT

// START-SETUP OMIT
func TestSine(t *testing.T) {
	const period = 1024
	amplitudes := []stream.Sample{stream.MaxSample, 10}

	for _, amplitude := range amplitudes {
		// END-SETUP OMIT
		// START-CONSTRUCT OMIT
		synth := NewSine(big.NewRat(period, 1), amplitude)
		st, err := synth.Play(0)
		if err != nil {
			t.Errorf("NewSine(%d, %d).Play(0) = _, %v", period, amplitude, err)
			continue
		}

		buf := make([]stream.Sample, period)
		st.Fill(buf)
		// END-CONSTRUCT OMIT

        if min, max := minSample(buf), maxSample(buf); min != -amplitude || max != amplitude {
            t.Errorf("range of NewSine(%d, %d).Play(0).Fill(<len = %d>) = (%v, %v);"+
                "want (%v, %v)",
                period, amplitude, period, min, max, -amplitude, amplitude)
        }

        if got, want := mean(buf), big.NewRat(42, 1); got.Cmp(want) != 0 {
            t.Errorf("mean of NewSine(%d, %d).Play(0).Fill(<len = %d>) = %v; want %v",
                period, amplitude, period, got, want)
        }
	}
}

descriptive failures

synths_test.go:82: mean of NewSine(1024, 32767).Play(0).Fill(<len = 1024>) = 0/1; want 42/1
synths_test.go:82: mean of NewSine(1024, 10).Play(0).Fill(<len = 1024>) = 0/1; want 42/1

The error message is exactly as clear as I wrote it to be, and I can predict its contents just by reading the test - no complicated nesting to worry about.

(It's still a good idea to try each failure message once, though - formatting directives are a bit subtle at times.)

The message includes all of the relevant parameters, what was expected, and what was actually got - and it's all in a reasonably Go-like syntax.

no fate but what we make

If we want to leave out information, we can elide it with _ in the failure message, just like we would when binding Go variables.

If we want to describe things with text, we can have them implement fmt.Stringer.

Or we can use %+v or %#v to print the internals in Go syntax.

Or we can simply replace that part of the message with some other description - such as <len = %d>.

state secrets

If we're working with a more stateful API, we might need to add a bit more context - for example, the calls to set up the thing being tested, or the contents of the database we're testing against.

t.Errorf("with db =\n%v\nStore(db, %v) = %v", db.Contents(), update, err)

We can even build up a description as a slice of strings, or emit it with t.Logf:

for _, action := range actions {
  if err := thing.Do(action); err != nil {
    t.Fatalf("thing.Do(%v) = %v", action, err)
  }
  t.Logf("thing.Do(%v)", action)
}

test functions are Go functions

If we're checking simple values, we can use ==.

For slices and larger structs, we can use reflect.DeepEqual.

If even that isn't enough, we can use any other Go function we want to check the output - such as (*big.Rat).Cmp in our synth test.

We don't need to implement custom matchers.

We can just call a Go function that tells us what we need to know.

what are we testing anyway?

Part of the purpose of testing is to check that the implementation is correct.

But it is even more important to check the API.

As you write tests, you should look for boilerplate, unnecessary state, and other sources of awkwardness.

With a complicated testing framework, these issues become much harder to spot: the less the test looks like real code, the less it will tell you about how real code will look.

Test code is code. Write it as code.

Thank you

Use the left and right arrow keys or click the left and right edges of the page to navigate between slides.
(Press 'H' or navigate to hide this message.)