A performance run finished with no pass rule agreed beforehand. What can its report honestly conclude?

## What a run without a rule actually produced A performance run with no pass rule agreed in advance has still produced something valuable: a **description** of what the system does under a set of conditions. What it has not produced is an outcome, because an outcome requires a rule, and there was none. The distinction is not pedantry. A description can be reused - another team can read it, a later run can be set beside it, and the numbers keep their meaning as long as the conditions travel with them. A judgement invented afterwards cannot be reused, because it encodes a choice made with the answer already on the screen. ## What the report may and may not say | The report may say | The report may not say | | --- | --- | | The workload applied: arrival rate, operation mix, data size, duration | That the system passed, or met expectations | | Where the timings were taken, and the intervals they cover | That the result is acceptable, or unacceptable | | The shape of the distribution, including its slow end | That a bound invented now was met | | What changed over the course of the run | That the feature is ready to release | That last row is the one people reach for. A run with no agreed rule cannot support a release decision on its own; that decision has its own criteria and its own owner, and offering an unjudged run as though it settled anything is how a number ends up quoted later as a commitment nobody made. ## Why a bound chosen afterwards is worth so little Three reasons, in increasing order of seriousness: 1. **It is fitted to the result.** Whoever picks it can see which values produce which answer. Even with entirely honest intent, the choice is anchored on what is visible. 2. **It cannot be falsified in hindsight.** Nobody can demonstrate that a different bound would have been chosen had the numbers been worse, so the pass says nothing about the system - only about the room it was decided in. 3. **It gets quoted later.** A recorded pass outlives every caveat around it. Six months on, the line in the report is read as evidence that the system met a requirement it was never checked against. ## The real output: the rule for the next run The most useful thing to write at the bottom of that report is the rule the next run will be judged against. The run just performed is the raw material for it: you now know roughly where the numbers sit, so you can set a bound that is meaningful rather than arbitrary - provided you write it down and agree it *before* the next run rather than after it. Settle all of it in one place: - **The measurement** - which quantity, for which operation, counted from which moment to which. - **The point in the distribution** the bound attaches to, and a cap on what lies beyond it. - **The window** the numbers come from, and the intervals they are reported in. - **The workload** the rule holds under, described in enough detail to be reproduced. - **The vantage point** the timings are taken from. - **Who may accept a breach**, and where that acceptance is recorded. - **The decision this run is meant to inform**, so the rule is aimed at something real. Using this run's numbers to place the next run's bound is legitimate and using them to declare this run a pass is not, and the difference is entirely the order of events. In the first case the bound is a commitment made while the next result is still unknown; in the second it is a description of a result already in hand, wearing the costume of a standard. ## Two objections worth answering out loud *"The numbers look fine, so surely we can call it a pass."* Fine was never defined, so the pass carries no content and cannot be repeated. The next run gets a larger dataset or a different mix, the same reasoning is applied to it, and a different word comes out. Nothing in that sequence is measurement. *"Then compare it against the earlier run instead."* A comparison requires the two runs to have been conducted under matching conditions, which is a substantial question in its own right, and even when the answer is yes, a comparison gives the direction of change rather than whether either run is acceptable. Direction with no agreed bound is still not an outcome. ## Saying it to whoever commissioned the run The honest sentence is short. "Here is what the system did under these conditions. We did not agree beforehand what would count as good enough, so I am not going to tell you now whether it passed - I would be choosing the bar with the result in front of me. Here is the rule I propose we agree, and I can re-run against it." That answer costs one run and buys every later run its meaning.