In benchstat's comparison output, what do the p-value, the n count, and a ~ in the delta column tell you?

## benchstat is running a hypothesis test The mental model that makes the output readable: benchstat holds two *sets* of numbers for each benchmark — say ten samples from `old.txt` and ten from `new.txt` — and asks a single question. *If these two sets had actually been drawn from the same underlying distribution, how surprising would a gap this big be?* The answer to that question is the p-value. The default test is the Mann-Whitney U test, a rank-based, non-parametric test. Non-parametric matters here: benchmark timings are not normally distributed — they have a floor and a long right tail, because a run can be slowed by an interrupt but never sped up below the work's cost. A test that assumed normality would be wrong about exactly the data you are feeding it. ## Reading each piece **The delta**, e.g. `-19.20%`, is how far the new column's central value sits from the base column's. It is the size of the effect, and it is the only part of the row that tells you whether the change is worth having. **`p=0.000`** is the significance. Small p means "a gap this large would be surprising if nothing changed". benchstat's default confidence level is 95%, so the cut-off is 0.05. It is *not* the probability that your change is good, and it is *not* the size of the improvement — the single most common misreading. **`n=10`** is how many samples per side went into the row. This is the honesty column. A stunning delta at `n=3` is an anecdote; a modest delta at `n=20` on a quiet machine is a result. **`~`** is printed instead of a delta when p exceeds the threshold. It means *we failed to detect a difference*, which has two very different causes: there really is no meaningful change, or your experiment was too noisy or too small to see one. A `~` is never proof of equivalence. **`± 3%`** describes the spread of the samples behind a column. Read it before you read the delta: if the base column's samples wobble by ±8%, a 5% delta was never going to be visible, whatever the p-value says. **`geomean`** is the bottom row: the benchmarks in the table combined with a geometric mean, so a change that helps some benchmarks and hurts others gets one overall number. A geometric mean is the right average for ratios — it does not let one benchmark that got 10x faster drown out five that got slightly slower the way an arithmetic mean would. It is only meaningful if the benchmarks in the table are all ones you care about; a suite padded with irrelevant micro-benchmarks produces a flattering geomean. ## The two failure modes **Chasing significance.** `-0.35% (p=0.001 n=20)` is a genuine, reproducible, statistically solid improvement that no user will ever perceive, and defending a complicated change with it is a bad trade. Significance answers "is it real"; only the delta answers "is it worth it". **Dismissing on a tilde.** A promising change that comes back `~` has not been refuted. Look at `n` and `±` first: raise `-count`, move to a quieter machine, and re-run before concluding the idea was worthless. ## A worked reading ``` Encode-8 1.203µ ± 2% 0.972µ ± 3% -19.20% (p=0.000 n=10) Decode-8 842.1n ± 1% 838.4n ± 2% ~ (p=0.353 n=10) ``` Encode is 19% faster, the samples were tight, and the result is very unlikely to be chance — believe it. Decode moved by a fraction of a percent and the test declined to call it; treat Decode as unchanged, and specifically do not report "Decode also improved slightly" in the pull request. ## Where the p-value stops helping The test only knows about the numbers it was handed. If both files were produced on the same busy shared machine, but the base ran while a build was hogging the cores and the new one ran after it finished, the test will happily report a large effect with a tiny p-value. Statistics protect you from random noise, not from a biased experiment.