Your organisation wants to track one safety benchmark's attack-success rate as a quarterly figure across model upgrades. What has to be frozen for that series to mean anything, and what silently breaks it?

A trend line is readable only if exactly one thing varied between points. In a quarterly ASR series the one thing is meant to be the model, and everything else in the instrument has to be pinned as hard as a dependency lockfile. ### Treat the instrument as a versioned artefact Vendor the behaviour list into your own repository at a fixed revision rather than pulling it from upstream at run time. Pin the harness version, the attack configuration and its budget, attempts per item and the aggregation rule, decoding settings (temperature, top-p, max tokens, seed where honoured), the target wrapper, and the ruling procedure including the judge's own weights or its exact prompt. Then record that whole tuple *with every data point*: ``` 2026-Q3 list=jbb@a1b4c9 n=25 agg=any-of-n attack=tmpl-v3 budget=1 turn target=chat-endpoint+sysprompt-v7 temp=1.0 judge=harm-clf@2026-02 ASR=11.4% ``` A point without its tuple cannot be re-checked, and in a series that runs for years you will need to re-check. ### The breakers that raise no error - **Item refresh.** The suite adds, edits or retires behaviours; your rate steps for population reasons and nothing logs it. - **Judge drift.** The classifier is retrained or the judge model behind a hosted API is updated underneath you, so quarters are scored by different rulings while the chart pretends they are one series. - **Harness defaults.** A tool upgrade silently changes a default temperature, max-token cap or retry count — retries in particular quietly change the effective attempts per item. - **Contamination.** A public list becomes widely mirrored and later models have effectively seen it. ### Contamination is the hardest one, because it flatters The failure mode is that the rate falls and everyone congratulates the safety work. Defend with a private held-out set of comparable items, written and maintained by your own team, run every quarter beside the public list. Divergence — the public line improving while the private line is flat — is the contamination signal. Keep the private items out of any external system, rotate a fraction of them periodically, and treat a leak of the private set as an incident, because it destroys the only control you have. ### What a quarter costs, and what that implies Each data point is a full sweep: `behaviours x attempts` generations plus a judging call per transcript, doubled if you run a private control list beside the public one — for a few hundred behaviours at a couple of dozen samples, tens of thousands of calls, hours of wall clock and a bill in the tens to low hundreds of dollars per model, per quarter. Small enough to run, large enough that nobody re-runs the whole history on a whim. That cost asymmetry drives one concrete design decision: **archive every transcript**. Re-scoring stored transcripts under a new judge costs only the judging half and needs no model access at all, so a judge change is recoverable; re-running old models is often impossible once a provider retires a version. Storage for a few years of transcripts is trivial next to the inference you already paid for. ### Re-baselining discipline When something must change — you adopt a refreshed list, or the judge is replaced — do not splice. Run the new instrument over one or two archived quarters to create an overlap window, publish both series through it, and annotate the changeover on the chart itself. Without overlap a step in the line is unattributable and the history becomes decoration. ### Where the number misleads The series can support a regression alarm: this quarter's model is materially worse on a fixed instrument than last quarter's, investigate. It cannot support an absolute safety claim, because the denominator is one curated list under one attack at one budget, and the figure would move under any other instrument. Say that in the chart caption, not a footnote — a single tracked percentage travels into slide decks stripped of its tuple faster than anything else a red team produces, and a quarter later someone will compare it to a vendor's published number. Finally, name an owner for the instrument, separate from whoever consumes the trend. That owner approves re-baselines, maintains the private control set, and refuses out-of-band changes during a reporting window — which is the only mechanism that actually stops a well-meaning upgrade from silently breaking a year of history.