What evidence should a Go benchmark speedup claim carry before you merge it, and should regressions fail CI?

## Two separate decisions This question hides two calls that get muddled: what a *person* must show to have a performance claim accepted in review, and what a *machine* is allowed to block a merge for. They have different economics and should be answered separately. ## The review bar The cost of the bar falls on every author, so it must be proportionate. Most changes claim nothing about performance and need nothing. The bar applies when a description asserts a number, or when a change is *justified* by performance — complexity accepted, an abstraction removed, a cache introduced. What such a claim should carry: - **The command.** Which benchmarks, what `-count`, what `-benchtime`, whether `-benchmem` was on. Without it the result is not reproducible and the reviewer cannot judge whether it had any power. - **A benchstat table, not two numbers.** With `n` and `p` visible. Two pasted per-operation figures are the artefact this whole discipline exists to reject. - **One machine, ideally interleaved.** Stated explicitly, because it is the assumption most often violated and never mentioned. - **A delta above the known noise floor of that machine.** Which presumes somebody has measured that floor by comparing a commit against itself. Publishing that number once, per benchmark machine, is one of the highest-leverage things a team can do here — it converts "is 6% real?" from an argument into a lookup. - **The benchmark itself, committed.** A win measured with a throwaway benchmark cannot be re-checked next quarter and cannot regress detectably. And what you deliberately do *not* require: a formal statistical case for every micro-refactor. If the bar makes ordinary work expensive, authors stop measuring rather than start. ## The gating decision Now the harder call. A blocking benchmark gate has a failure mode that most quality gates do not: it can be *wrong at random*. A flaky red build teaches the team that red means nothing, and that lesson transfers to the gates that were telling the truth. So the question is not "would a gate be nice" but "can this gate be right often enough to keep its authority". That depends almost entirely on the hardware you can give it. **On shared runners**, timing gates cannot be right often enough. The machine's own variation is frequently larger than the regressions worth catching. But not everything a benchmark reports is noisy: allocation counts and bytes per operation are deterministic for a given input. A gate that fails when `allocs/op` on a listed hot-path benchmark rises above a recorded value is reproducible, explains itself precisely, and catches a large fraction of real regressions — accidentally boxing a value, an added copy, a logging call in a loop. That is the gate I would build first. **On dedicated, quiet hardware**, a timing gate becomes defensible: run it nightly rather than per pull request, alert rather than block for the first quarter, set the threshold well above the measured A/A noise floor, and only then let it block. Bisect-friendliness matters more than immediacy here — knowing which of yesterday's twenty merges cost you 8% is usually enough. **Whatever the gate, it must show its work.** A failure that prints the benchmark name and the benchstat table is actionable. A failure that says "performance check failed" gets a retry, then an exemption, then deletion. ## Owning the tradeoff, and being overruled This is where the call stops being technical. A tech lead who says "the benchmark gate has cost us four hours of retries this month, turn it off" is making a legitimate argument about throughput, and the honest answer is usually to concede the gate and keep the signal: move to alert-only, narrow it to the two benchmarks that guard the actual hot path, or spend the money on a machine that makes the gate trustworthy. Insisting on a gate that the team has already learned to ignore is worse than losing the argument, because it leaves everyone believing they are protected. The other cost worth naming out loud: benchmark suites accumulate. Every benchmark added to a gate is a permanent tax on every merge, and most of them do not guard anything anyone notices. Keeping the gated list short, tied to code paths with a stated performance requirement, is what keeps the whole arrangement affordable. ## The last question Before any of this, ask whether the benchmark represents something users experience. A suite that gates on micro-benchmarks nobody's traffic resembles will happily block a change that would have made production faster. The evidence bar and the gate are both instruments for a purpose, and the purpose is the service's behaviour, not the suite's numbers.