Your organisation wants to gate model releases on one robustness number from an adversarial library — a mean perturbation size over only the examples an attack flipped. What breaks when that number is tracked release over release, and what would you gate on instead?

**The three failures, in the order they will actually bite.** 1. **Non-monotonicity.** A gate must move in one direction with the property it protects; otherwise crossing it means nothing. This number does not. Its population is the set of rows the attack flipped, which is selected by each release's own weaknesses, so an improvement and a regression can both push it up or down. A model that gets harder to break may lose only borderline points, whose small distances *lower* the mean; a model that regresses may start losing far-from-boundary points, whose large distances *raise* it. A gate that can be tripped by improving the model, and passed by degrading it, will be argued around within two release cycles and then quietly ignored. 2. **The degenerate case.** When a release flips nothing, the helper divides by an empty success set and returns `0.0` — a guard-clause placeholder, not a measurement. On a scale where small means fragile, your best release scores worst in the history. Someone then adds a special case for it, and the special case becomes the piece of gate logic nobody remembers is there. 3. **Goodhart pressure.** The moment a number blocks a release it becomes an optimisation target, and the model is the hardest of its inputs to change. The attack configuration, the example set, the baseline filter and the library version are all levers that move the number without touching the model at all. If they are not pinned and reviewed as code, the gate is decorative — and it will be tuned, not maliciously, but by an engineer under deadline who genuinely believes the attack settings were too aggressive. **What a defensible gate looks like.** - **Frozen inputs.** One versioned example set; the attack family, its search settings and its perturbation budget pinned in configuration; the baseline-correct filter pinned; the library version pinned. Changing any of them is a reviewed change with a re-baseline, never a run-time flag. - **A rate, not a conditional mean.** Attack success rate at the pinned budget, with the denominator being every eligible row. Add a second, larger budget if you care about a stronger threat model — two rates read fine on one slide and bound the model between two stated adversaries. - **Counts always carried.** `n_evaluated`, `n_eligible_after_baseline_filter`, `n_flipped`. Without them nobody can distinguish a real improvement from a shrunken evaluation set, which is the single easiest way to fake a passing gate by accident. - **Direction decided in advance.** State the threshold and the consequence of crossing it before the release, so the gate is policy rather than an argument reopened every time it fails. **What it costs to run this way, and why that shapes the design.** Every release now pays for a full attack sweep over the frozen set: for an iterative gradient attack that is tens of forward and backward passes per row, so a few thousand rows is minutes to hours of accelerator time per release — schedulable, but real, and it is the reason people are tempted to shrink the set or weaken the attack. Every pin change costs a re-baseline: you must re-run the *previous* release under the new configuration too, or the series has a discontinuity nobody can interpret. Budget for that explicitly, because an un-budgeted re-baseline is how a pin quietly stops being a pin. And if the gate targets a hosted, metered endpoint rather than a local model, the run has a per-release invoice attached, which changes the argument from engineering to procurement. **Where the gate still misleads even when it is built correctly.** It measures one attack family, at one budget, on one frozen set, against a baseline-filtered population. It is an upper bound on that model's resistance to *that specific procedure* — nothing more. A green gate is not a safety statement, it is not evidence about attack families you did not run, and it says nothing about rows outside your frozen set or about drift in production inputs. Because a green result is exactly the artefact stakeholders over-read, I write that limitation into the release note itself, next to the number, rather than into an appendix: this release was not broken more than x% of the time by attack F at budget B over n rows, and that is the entire claim. The failure this whole discipline exists to prevent is a number that outlives the conditions under which it was measured.