Your organisation publishes a quarterly safety attack-success rate for every model it ships, computed by an automated harm judge, and better judges keep appearing. How do you decide when to change the judge, and what do you owe readers of the historical series?

**Frame the decision correctly.** A quarterly published attack-success rate is a measurement instrument with two obligations that pull against each other: **comparability** across quarters, and **fidelity** to the current best understanding of what a successful attack is. Upgrading the harm judge buys fidelity and spends comparability, and unlike a model change the spend is invisible from outside — the number moves, nothing in the release notes says the ruler changed, and readers attribute the movement to the product. **The policy I would run.** 1. *The judge is part of the metric definition.* The series is named for its scorer version, exactly as a schema is. Changing the scorer is a metric revision with a version bump and a changelog entry, never an implementation detail someone lands in a scoring script. 2. *Archive from quarter one.* Prompts, full untruncated completions, decoding parameters, seeds, attempts per behaviour and the reduction rule, retained for the life of the series. This is the decision that determines whether an upgrade is ever possible, and it has to be made before the first publication — you cannot retroactively keep transcripts you threw away. The storage is trivial (kilobytes per attempt, single-digit gigabytes per year of quarterly runs); the retention *policy* is the hard part, especially where transcripts contain harmful content that other policies want deleted. Settle that conflict deliberately, in writing, up front. 3. *Re-score the archive on upgrade.* New judge, entire history, both series published over at least one overlap period, with the quarter's movement explicitly decomposed into judge effect and model effect. 4. *Publish the instrument, not just the number.* Judge identity and version, measured false-positive and false-negative rates, the size and provenance of the adjudicated sample behind them, and a minimum resolvable difference below which quarter-over-quarter movement is not narrated as change. 5. *Prefer a pinnable scorer for the headline series.* A hosted model behind an endpoint can be updated without notice, which is a silent drift channel in the metric. Run a locally hosted classifier or a snapshot-pinned model for the series; use the better hosted judge for exploration and for the periodic audit. 6. *Drift canary.* A frozen, human-adjudicated transcript set re-scored every cycle. If its labels move on unchanged inputs, the instrument moved, and you learn that before the headline does. **What the policy costs.** Steady state is cheap: the canary is a few hundred transcripts per cycle, and the archive is negligible storage. The bill lands on an upgrade, because re-scoring the whole history scales with quarters times transcripts — eight quarters at twenty thousand transcripts is a hundred and sixty thousand scoring calls, which is a real GPU day or a real invoice, plus the engineer time to re-run pipelines whose code has since moved on. That cost is the reason organisations skip step 3, and skipping it is precisely what turns the series into fiction. Budget the re-score into the upgrade decision rather than discovering it afterwards. **Where the number misleads.** The headline failure is the silent swap: the scorer is upgraded between quarters, the rate drops three points, and the release notes read it as a safety improvement. The seductive shortcut — apply a fixed offset to past quarters so the line joins up — is worse than doing nothing, because the two judges disagree unevenly across behaviours, attack families and models, so a single offset fabricates a correction no data supports. Equally misleading in the other direction: an unchanged judge with a demonstrated blind spot on an attack family that is now in scope reports a flat, comparable, and entirely uninformative series. **Governance.** Whoever owns the judge owns the metric. If the team optimising the models also owns and tunes the scorer, the number is captured; separate the ownership, or at minimum require that judge changes are reviewed and dated by someone outside model development. **When I would and would not change.** I would refuse a mid-series swap for a marginal accuracy gain when the archive cannot be re-scored — the comparability loss exceeds the benefit. I would change immediately, and take the discontinuity, when the incumbent judge has a demonstrated systematic blind spot on an in-scope attack family, because a series that cannot see a real risk class is worse than a break in the line. Either way the decision is published with its reasoning, so a reader can tell an instrument event from a product event.