Optimising with Evidence

## Why two numbers are not a comparison A Go benchmark prints a time per operation for one binary, on one machine, at one moment. Run the identical binary again and the number moves: CPU frequency scaling, co-tenant processes, memory layout, and garbage-collection timing all shift it by a few percent and sometimes much more. So comparing one number from before a change against one number from after tells you almost nothing — the difference you see is *the change plus the noise*, and a subtraction cannot separate them. `benchstat`, from the `golang.org/x/perf` repository, does that separation. It reads *samples* — repeated measurements of the same benchmark — from two or more files, and reports per benchmark how far the metric moved and how confident it is that the move is real. ## Step one: produce the samples One repetition of a benchmark produces one output line, which is one sample. `-count=N` repeats every selected benchmark N times, so `-count=10` gives ten samples: ``` go test -run=^$ -bench=Encode -count=10 -benchmem > old.txt ``` - `-run=^$` is a regular expression that matches no test name, so only benchmarks run. Without it the package's unit tests execute first and add noise and time. - `-bench=Encode` selects benchmarks whose name matches; `-bench=.` selects all of them. - `-count=10` produces the samples. Ten is the usual starting point; noisy machines need more. - `-benchmem` adds the bytes-per-operation and allocations-per-operation columns, which are worth having on every comparison because they are far steadier than time. The file is just the raw text output. benchstat parses the `goos:`, `goarch:` and `pkg:` header lines and the benchmark result lines, and ignores the rest. Then check out or apply the change and run **the same command** into `new.txt`. ## Step two: compare ``` benchstat old.txt new.txt ``` You get one row per benchmark, a column per input file, a `vs base` percentage delta, and a `p=… n=…` annotation. `~` in the delta column means the test could not distinguish the two sample sets. The bottom `geomean` row combines the table into a single overall figure, which is useful when a change moves several benchmarks in different directions. benchstat accepts one file (it just summarises it) or more than two (each extra file becomes another column compared against the first). ## What keeps the two files comparable - **The same machine**, ideally an idle one. Two files produced on different hardware are not a comparison of your change. - **The same toolchain version**, since compiler changes move benchmark numbers on their own. - **The same flags**, especially `-benchtime` and `-benchmem`. - **The same benchmark names.** benchstat pairs rows by name; if you rename `BenchmarkEncode` while you are at it, you get two unpaired rows and no delta. - **The same GOMAXPROCS.** The `-8` suffix on `BenchmarkEncode-8` is the GOMAXPROCS value the benchmark ran with, and it is part of the row's identity — samples taken on a 8-core and a 16-core machine will not pair. ## Installing it benchstat does not ship with the `go` command: ``` go install golang.org/x/perf/cmd/benchstat@latest ``` It then lives in `$(go env GOPATH)/bin`. ## What benchstat will not do for you It reports *whether* something moved, never *why*. It cannot tell you whether the benchmark resembles production traffic, and it cannot rescue an experiment that had too few samples or ran on a machine that was doing other work. Those judgments stay with you; benchstat only makes sure you are not reading noise as a result.