Should a Rust project let criterion benchmarks block merges in CI, and how would you keep runner noise from failing good pull requests?
answer
- criterion reports but never fails
- change files under target/criterion
- test mode on every pull request
- same machine, same job, base then head
- thresholds from runs of unchanged code
basics
~20 sGate only a few critical benchmarks, on a quiet dedicated machine, comparing base and head in one job against thresholds above measured noise; run every benchmark once per pull request. Criterion itself never fails a regressed run.
solid answer
~50 sStart from what criterion gives you: it printed `Performance has regressed.` for a +746% change and still exited with status 0, so any gate is code you write, usually reading `target/criterion/<group>/<bench>/change/estimates.json`. Next, the defaults are too tight for shared runners: on a laptop, an unchanged `fnv1a` came back as `Performance has improved.` at −4.4%. A workable policy has layers. Every pull request runs the benchmarks in test mode through Cargo's test command, which calls each one once without measuring and catches broken benchmarks cheaply. A small set of hot paths is timed on dedicated hardware, with the merge base saved as a baseline and the head compared to it in the same job, failing only when the whole interval clears a threshold set from runs of unchanged code. Everything else is trended, not blocking, and someone owns the false alarms.
code
plaintext · 6 lineschecksum-64KiB/adler32 time: [142.56 µs 144.01 µs 145.70 µs]
thrpt: [428.98 MiB/s 433.99 MiB/s 438.42 MiB/s]
change:
time: [+729.95% +746.13% +764.43%] (p = 0.00 < 0.05)
thrpt: [−88.432% −88.182% −87.951%]
Performance has regressed.go deeper
Know that criterion compares runs but does not fail a build on a regression, and that benchmark results depend on the machine they ran on.
Explain how a gate reads criterion's change files, why the lower bound of the interval is the number to check, and what test mode is for.
Describe running base and head in one job on one machine, deriving thresholds from runs of unchanged code, and why stale change files must be cleared.
Own the policy: which benchmarks block, which only trend, what hardware it needs, who answers a false alarm, and how a deliberate slowdown is approved.
## What criterion gives a CI pipeline A lead deciding on a benchmark gate should first know what the tool does and does not do: - **No failing exit status for regressions.** Criterion 0.7 prints `Performance has regressed.` and still exits with status 0. A run compared against a saved baseline with a +746% change returned 0. The process fails only for errors, such as `--baseline` naming a baseline that does not exist, which panics. - **Machine-readable results.** Each compared benchmark gets `target/criterion/<group>/<bench>/change/estimates.json`, holding the relative change of the mean and median with their confidence intervals (7.29 means +729%). A gate script reads those. - **Stale files stay.** Criterion does not delete an old `change` directory, so a gate should run in a clean target directory or it may read a comparison from an earlier run. - **A cheap test mode.** Running the bench targets through Cargo's test command makes criterion call each benchmark once and print `Testing ...` then `Success`, with no measurement. - **Adjustable thresholds** through `--noise-threshold` and `--significance-level`, or on a `Criterion` or `BenchmarkGroup` in code. - **Time cost.** With the defaults each benchmark spends at least 3 s warming up and about 5 s sampling, so a suite of 200 takes about half an hour before analysis. ## Why the defaults are wrong for a gate Criterion's default noise threshold is 1% and its significance level 0.05. Those are good for a developer comparing two runs on one machine. They are not a merge gate: an unchanged function measured on a laptop moved by +2.2% in one comparison and −4.4% in another, and the second was reported as `Performance has improved.` A gate at the defaults would fail good pull requests for the machine's sake, and developers learn to rerun or ignore it. ## The options a lead chooses between | Policy | What it catches | Cost | False alarms | |---|---|---|---| | No benchmarks in CI | nothing | none | none | | Test mode on every pull request | benchmarks that no longer compile or panic | seconds | none | | Timed, report-only trend | gradual drift, seen after the fact | a runner and a dashboard | harmless, but ignored if noisy | | Timed gate on shared runners | large regressions | minutes per run | frequent, erodes trust | | Timed gate on dedicated hardware, base and head in one job | regressions above the measured noise | a machine, maintenance | rare if thresholds come from data | ## A workable policy 1. **Run every benchmark in test mode on every pull request.** Benchmarks rot like any other code; this keeps them compiling and running. 2. **Pick the few benchmarks that guard a real promise** (a parser's throughput, a checksum on the hot path) and gate only those. 3. **Measure base and head on the same machine in the same job**: check out the merge base and save it with `--save-baseline main`, then build the head and compare with `--baseline main`. A baseline saved weeks ago on another runner carries that machine's state into the verdict. 4. **Set thresholds from data.** Run the same commit against itself several times on the gate machine and set the limit above the spread you see; criterion's `--noise-threshold` raises its own verdict, and the gate script applies the same limit to the lower bound of the interval. 5. **Fail on the lower bound**, not the point estimate, so a gate trips only when the whole interval clears the limit. 6. **Give the gate an owner and an override**, so a deliberate trade (slower but correct) can merge with a recorded reason. ## Costs to weigh - **Hardware.** A quiet, dedicated runner with frequency scaling under control is the difference between a gate and a coin flip. - **Wall-clock time** grows with every benchmark added, which pushes teams toward a small gated set and a larger trended one. - **Trust.** One week of false failures teaches a team to rerun until green; a gate nobody believes is worse than none. - **Coverage gaps.** A gate only guards the benchmarks it runs, so it complements profiling and production metrics rather than replacing them. There is no universal answer. A library whose value is speed may justify dedicated hardware and a strict gate on its core paths; a service team may get more from test mode plus a trend that someone reads every week.
- Why compare base and head in the same job instead of against a stored baseline?A stored baseline froze one machine's state: its kernel, firmware, temperature and neighbours. Measuring the merge base with `--save-baseline main` and then the head with `--baseline main` on the same machine, minutes apart, cancels much of that drift, so the threshold can be tighter without more false alarms. It doubles the timing cost, which is one more reason to gate only a few benchmarks.
- How would you pick the threshold for a gated criterion benchmark?Run the same commit against itself repeatedly on the gate machine and read criterion's change intervals; the gate's limit sits above the largest spread you saw, not at the 1% default. On a laptop an unchanged function moved by −4.4% and +2.2% across runs, so 1% would fail good pull requests. Revisit the limit whenever the gate machine changes.
- What does running the benchmarks in test mode on every pull request buy you?It proves every benchmark still compiles and runs to completion without spending time measuring: criterion calls each benchmark once and prints `Testing` and `Success`. A refactor that breaks a benchmark is caught the day it happens, instead of when someone needs the benchmark for a performance investigation.
- Criterion fails the CI job when it prints Performance has regressed.
- The default 1% noise threshold is a sensible merge gate on shared runners.
- A baseline saved last month on another runner is fine to compare against.
- Every benchmark should block merges, or regressions will slip through.
- Benchmarks have no place in CI at all because runners are noisy.