Before you call a difference between two performance runs a regression, what must you measure first?

## Why the spread has to exist first A performance figure is not a constant. Run the same build against the same data on the same machine twice and the numbers differ — because of scheduling, cache state, disk and network variation, shared infrastructure, background activity, and the randomness inside the applied workload itself. That variation has a size. Until you know it, every comparison you make is a claim without a scale. The consequences of skipping it run in both directions. A team without a spread figure chases differences that were never there, burning days on a *regression* that repeats would have dissolved. The same team, a month later, waves through a genuine loss because *it is only a few percent* — with no idea whether a few percent is inside the noise or five times it. Both failures come from the same missing number. The figure is also the input to almost everything else in this area: it sets the smallest change that can be detected at all, it sets any regression alarm level worth having, and it is the evidence that a shift in level was reproduced rather than observed once. ## How to measure it 1. **Fix the configuration completely.** One build, one dataset snapshot, one environment specification, one warm-up procedure, one workload definition, one measured window. Nothing is under study; you are measuring the apparatus. 2. **Repeat it several times.** Three is a hint, five is a working minimum, more is better for a noisy environment. Fewer repeats give a spread figure that is itself noisy. 3. **Vary what you cannot hold, deliberately.** Run some repeats back to back and some across separate provisionings of the environment, on different days or times. Back-to-back repeats measure only the variation *within* one setup and will flatter you; the variation that bites in practice includes being placed on a different physical host. 4. **Record the individual figures, not only a summary.** You want to see whether the repeats cluster with one outlier, or spread evenly, or fall into two groups — those three shapes have different causes and different remedies. 5. **Express the dispersion.** The range across repeats relative to their middle value is enough for most teams and is easy to explain. A standard deviation is fine if the repeats look unimodal. Publish it beside the result, in the same units and framing as the comparison it governs. 6. **Do it per figure.** Completed work over a long hold is usually steadier than a high percentile, and the slow end of the latency distribution is usually the noisiest thing you measure — which is unfortunate, because it is often the thing you care about most. ## What the number is worth, and when it expires | Situation | What the spread figure tells you | | --- | --- | | Repeats cluster tightly | Small differences are meaningful; you can detect fine changes | | Repeats spread widely | Only large differences are detectable; fine changes need more repeats or a quieter environment | | One repeat far from the rest | Something intermittent is present; investigate it before trusting any comparison | | Repeats fall into two clear groups | The apparatus has two states — placement, cache warmth or a configuration that is not fully pinned | A spread figure describes one apparatus at one time. It expires when the environment changes shape, when the dataset is refreshed, when the workload definition is edited, or when the infrastructure underneath is re-provisioned onto different hardware. Treat it as a measurement with a date, re-take it periodically, and be suspicious when it improves dramatically without explanation — that usually means the repeats stopped being independent. ## Common ways this goes wrong - **Taking the best run.** The fastest of five is not a reference; it is the left tail of your noise, and comparing a later best-of-five to it is comparing two extremes. - **Repeating without re-provisioning.** All-back-to-back repeats systematically understate spread and produce alarm levels that trip in real use. - **Reusing a spread figure across metrics.** Applying the completed-work spread to a high percentile understates the noise where it is worst. - **Treating the gap between two builds as the spread.** That gap is the thing under test, not the apparatus's variability. - **Never refreshing it.** A figure measured on last year's environment says nothing about this one. The habit that makes this stick is publishing the spread beside every reported result, so that any reader — including the version of you reading it next quarter — can immediately see whether a difference of a given size means anything at all.