Before a performance run starts, what must its pass rule fix for the run's verdict to mean anything?

## What a pass rule is, and why it must exist first A **pass rule** is the sentence that turns a performance run's numbers into a verdict, written down and agreed before the run is started. It is not the product requirement it descends from, and it is not a promise to anyone outside the team. Its single job is to be **decidable**: two people reading the same result document must reach the same verdict without negotiating. It has to exist beforehand because a run does not produce *a* number. It produces a cloud of them - thousands of timings, split by operation, spread across intervals, taken at several points in the system. Almost any story can be told from that cloud once it exists, and fixing the rule first removes the freedom to pick which one. ## The five parts | Part | The question it settles | What leaving it open licenses | | --- | --- | --- | | Measurement | Which quantity, for which operation? | Quoting whichever operation happened to be fast | | Distribution point | A central measure, a high percentile, the worst observation? | Retreating to the central measure when the slow end looks bad | | Window | Which stretch of the run, reported in which intervals? | Trimming away the stretch that held the bad minutes | | Workload | Which arrival rate, operation mix and data size? | Passing by driving the system more gently than intended | | Vantage point | Where is the timing taken? | Reading whichever of two honest measurements is friendlier | 1. **The measurement.** "Response time" is not yet a measurement. Response time *of which operation*, counted from which moment to which moment? A rule that binds one figure over a mixed workload is satisfied by a flood of cheap reads while the operation anyone cares about is slow. 2. **The distribution point.** A bound has to attach to a stated place in the spread of results - the middle, a high percentile, or the worst observation. Those disagree by design, and a rule that does not say which one it means is not yet a rule. 3. **The window.** Numbers must come from a named stretch of the run and be reported in stated intervals, because the interval size changes what a breach even looks like: one bad thirty-second stretch disappears inside a ten-minute interval and dominates a one-minute one. 4. **The workload.** A time bound is meaningless without the load it holds under. Arrival rate, the mix of operations, the size and shape of the data, and the number of concurrent callers all belong in the rule, because every one of them moves the number. 5. **The vantage point.** The same call has several honest durations: timed by the caller, timed at the edge of the service, timed around the handler inside it. They differ by connection setup, by waiting before the work starts, and by any retry. The rule must name which span it bounds. ## Two clauses teams forget - **The number with its unit, written once.** "Under 400 ms" and "under 0.4 seconds" are the same rule; "under 400" is not a rule at all, and a figure each reader recomputes is a figure each reader can recompute favourably. - **Who may accept a breach, and how.** Runs breach for reasons everybody agrees are irrelevant. The rule should name a person who may accept a breach and require that the acceptance be recorded with a reason. An accepted breach with a name on it is a decision; an argument after the fact is not. ## A worked example Suppose a team agrees, before starting: the measurement is the response time of the *submit order* operation; the bound is 400 ms at the 95th percentile with no one-minute interval above 1200 ms; the window is the thirty minutes after load has stabilised, reported in one-minute intervals; the workload is 200 arrivals per second at seven browse calls per submit against a two-million-item catalogue; timings are taken by the caller that issued the request; and the service owner may accept a single breached interval, recorded with a reason. Nothing there is clever. Its virtue is that when the run finishes, the verdict falls out of the document rather than out of a meeting. ## What an unstated part licenses Every gap gets filled in afterwards, by whoever has the strongest interest in a particular outcome: - No workload named, so the run is repeated at a gentler rate until it passes. - No distribution point named, so the central measure is quoted and the slow end goes unmentioned. - No window named, so the report covers the calm stretch and stops there. - No vantage named, so the internal timings are shown and the caller-side ones are not. - No breach owner named, so the breach is explained away rather than accepted, and nobody records that it happened. None of those require dishonesty. They are what reasonable people do with an ambiguous rule and a deadline, so the fix is structural rather than moral: fix the rule while the numbers do not yet exist and the temptation never arises. Worth saying plainly, too - this is a rule for one run and nothing more. A long-lived reliability commitment about live traffic is a different subject with different owners.