Run Comparability

## Comparability is a property of a pair, not of one run A performance run measures a system under a particular set of conditions. Its numbers describe that combination, not the system in the abstract. The moment two runs are placed side by side and the gap is read as *the change made it slower*, an experiment is being claimed: one variable moved, everything else was held. If in fact three things moved, the gap is real but **unattributable** — it is the sum of every uncontrolled difference, and no statistic recovers the parts afterwards. So comparability is not a quality a single run can have. A run is comparable *to another run*, or it is not. This also means the axes are only worth holding pairwise: you do not need a perfect environment, you need the *same* environment twice. A distinction worth drawing early: **comparability is not realism**. Two runs on a modest environment can detect a relative change between them perfectly well, even though neither says much about production capacity in absolute terms. Realism governs how far a result generalises; comparability governs whether a difference means anything at all. Teams often throw away a valid regression signal on the grounds that the environment is not production-like, which confuses the two questions. ## The axes that have to be held | Axis | What it covers | What moves if you do not hold it | | --- | --- | --- | | Build and settings | The exact artefact plus pool sizes, feature switches, logging verbosity | A single verbose logging setting can move the slow end of the latency distribution | | Dataset | Row counts **and** distribution: key skew, how many rows a typical request touches | The plan the data store chooses changes as data grows, with no code change at all | | Environment | Machine class, storage, network path, and whatever else runs on the same host | A neighbour's overnight job arrives in your results as a regression | | Warm state | Whether caches, connection pools and lazily initialised paths were in the same condition | A cold start compared against a warm reference looks like a large loss | | Applied workload | Operation mix, arrival pattern, concurrency, think time, hold duration | A slightly different mix changes the average cost of a request | | Dependencies | Whether downstream systems were real, shared or substituted, and in what state | A shared downstream under someone else's load becomes your slowdown | | Measurement point | Where the timing was taken and what it includes | A client-side figure and a server-side figure are different quantities, not two views of one | ## What the report has to state The axes only help if the next person can check them, which is a records problem rather than an engineering one. A run report that will still be usable a quarter later carries: 1. The **build identifier** and the settings that differ from the shipped defaults. 2. How the **dataset** was produced, how large it was, and what shape it had. 3. The **environment specification** and what shared it during the measured window. 4. How the system was **warmed**, and for how long, before measurement began. 5. The **workload definition** — mix, arrival pattern, concurrency, hold duration. 6. The **measurement point** and the window the figures summarise. 7. The date, and the **run-to-run spread** if repeats were executed. A figure without that record is not a reference. It is a memory, and a quarter later nobody can tell whether a new run matches its conditions or whether the old machine even exists any more. ## Where comparability is lost quietly None of these announce themselves; each has retired a perfectly good comparison: - A dataset that grew between runs because the earlier run left its rows behind. - An environment re-provisioned onto a different machine generation with the same label. - A default that changed inside a dependency rather than in your own build. - Repeats taken at different times of day on infrastructure shared with other teams. - A workload definition edited *slightly* to work around a scripting problem mid-campaign. - A report that records the numbers but not the conditions that produced them. The cheap discipline is to emit a **run manifest** with every result, generated rather than typed, and to refuse to compare two runs whose manifests differ on an axis nobody intended to change. That refusal is not pedantry: it is the only thing standing between a team and a quarter spent chasing a regression that was a neighbour on a host. Finally, keep separate the two things a run's numbers get compared to. Comparing a run to an **earlier run** asks whether something changed; comparing it to an **agreed target** asks whether it is good enough. The axes above make the first comparison possible. They do not, on their own, make either verdict.