A quarter after your baseline, you re-run the same harmful-behaviour benchmark with the same prompts against the same hosted endpoint alias, and the attack-success rate drops noticeably. How do you establish whether the model behind the alias changed, your harm judge changed, or it is run-to-run variation?

### The rule Never compare two aggregate numbers when you can hold components fixed and compare one at a time. A moved attack-success rate has three ordinary suspects — the judge, your harness, and sampling noise — and only after all three are excluded does the fourth, the served system, become the explanation. The steps below are ordered by cost, cheapest first, which is also roughly the order of how often each turns out to be the culprit. ### Step 1 — re-judge the past (cheapest, most productive) Take the **stored baseline transcripts** and score them with the current harm judge. The prompts and responses are frozen text, so the model is held literally constant; any difference between this re-scored figure and the original baseline rate is judge drift, isolated exactly. Cost: judge calls only, no target-endpoint calls — for a thousand stored attempts that is minutes of compute and a few dollars, an order of magnitude cheaper than re-running the suite. This is the single strongest reason to retain transcripts, and it explains a surprising share of apparent drift, because harm judges are retrained and their thresholds are tuned. ### Step 2 — diff the manifests Compare the two run manifests field by field: dataset revision, behaviour subset size, temperature, `max_tokens`, attempts per behaviour, the success rule, prompt-template hash, system-prompt hash, and error accounting. The frequent culprit is the last one. If the earlier run dropped timed-out attempts from the denominator and the later run counted them as non-hits, the rate falls for a purely clerical reason and looks exactly like a safety improvement. Cost: near zero — this is a text diff, and it should be done before anyone looks at the rates at all. ### Step 3 — measure your own noise Repeat the current run two or three times under identical settings and look at the spread of the resulting rates. At temperature zero the spread is small but not zero — batching, load balancing across serving replicas and non-deterministic kernels all leak in — and at higher temperature it can be wide enough to swallow the quarter-over-quarter gap entirely. You want the *observed* spread as a local comparison band, not a textbook confidence interval. Cost: N times the full suite. Three repeats of a 200-behaviour, 5-attempt suite is roughly 3,000 target completions plus 3,000 judge calls, so this is the step where money starts to matter; a subset of behaviours is a legitimate economy if the subset is fixed in advance. ### Step 4 — attribute the residual, and know what it still bundles Whatever gap survives the first three steps belongs to the served system. That phrase bundles two different things an end-to-end rate cannot separate: retrained or re-tuned **model weights**, and **provider-side filtering** — an input or output classifier in front of the model that can block a request before the model sees it. Read transcripts rather than the aggregate to split them. A filter tends to emit a uniform, templated block message, a distinctive error code or a truncated empty completion; a retuned model produces varied, in-character refusals that engage the substance of the request. Asking the provider whether the alias was re-pointed is worth doing, but plan for silence. ### Step 5 — the control run, if the budget allows If the provider still serves a pinned snapshot matching your baseline date, run the current harness and current judge against that too. Same instrument, old weights: any gap between it and the alias is the served-model change, cleanly attributed. It costs one extra suite's worth of queries and is the closest thing to a controlled experiment available on a hosted endpoint. ### Where the number misleads Three specific traps. **Directional bias**: a fallen rate is welcomed and never investigated, so harness bugs that suppress successes — a mis-templated prompt the endpoint refuses for the wrong reason — are recorded as safety wins. **False precision**: a change from 14% to 11% sounds decisive until three repeats show a spread of four points. **Attribution by adjacency**: the provider happened to announce an update that month, so the movement is credited to it, when the judge revision landed the same week. Write the residual up as "the served system changed, with judge, harness and sampling excluded", and say plainly that the aggregate cannot distinguish model tuning from a filter.