An Attacker Who Knows

### What the number actually is Robust accuracy is the share of a test set a model still classifies correctly when an adversary is allowed to modify each input inside a stated budget. Two things determine that number, not one: the model-and-defense being measured, and **the attack that was run against it**. A published attack suite fixes the second half in advance, with searches designed for plain, undefended models — usually written before the defense under test was invented. The number that comes out is a joint property of your system and somebody else's assumptions. ### Two explanations for a high score, and they are not equally good news When a defense scores well against a stock suite, exactly two things could be true. 1. Inputs that the defended system reads wrongly, inside the stated budget, are genuinely hard to find. 2. Such inputs are still there, but the stock search cannot locate them because the defense perturbed the mechanics of searching — a randomization stage makes the signal the search follows noisy, a non-differentiable preprocessing step means there is no usable derivative to follow, an ensemble vote flattens the quantity the search was reading, a detector rejects the exact shape the suite happens to produce. The second case is common enough that the field has diagnostic signals for it: an attack given weaker access outscoring one given stronger access, or an attack with its budget removed entirely still not driving the model to total failure. Neither of those is possible if the search is working. A stock number cannot distinguish case 1 from case 2, because in both cases the same thing is observed — the suite returned without a break. ### Obscurity, measured What an unaware attack really measures is the *gap between the attacker's assumptions and your system*. That gap is real, and in the field it does buy time. But it is not a property of the model, it is not something you can promise a customer, and it shrinks every time your design is documented, inferred from behaviour, or reconstructed from a shipped artifact. That is why the adaptive convention grants the adversary the design rather than waiting for them to obtain it: a number that improves when your documentation gets worse is not measuring the defense. ### The evidence is asymmetric, which is why the burden sits with the claimant A successful break is decisive: something exists, someone found it, the claim is over. A failure to break is bounded by whoever tried, how hard, and from what vantage. This asymmetry has a direct consequence: producing the strongest defense-aware attack is the *author's* job. A reviewer who declines to spend a week attacking your design has not endorsed it, and a suite that never knew your design existed has not either. The reverse direction is worth stating too. If a generic, defense-unaware attack *does* break the model inside the stated budget, the claim is refuted immediately, and no adaptive work is required. Stock suites are weak evidence of strength and strong evidence of weakness — useful as a cheap screen, useless as a warrant. ### Why this rule exists at all This is the central methodological finding of the whole defense literature, and it was learned the expensive way: a long run of published defenses reported large robustness gains, and a large fraction of them were reduced to roughly the undefended level once somebody wrote an attack with the defense's description in hand. In almost none of those cases was the original evaluation dishonest. The evaluations ran real attacks and reported real numbers; the attacks simply had not been written against the thing being measured. ### What it looks like in a real engagement Take a static malware classifier shipped inside an endpoint agent, deciding block-or-allow at execution time. The vendor whitepaper describes the defense fully — a randomized feature-hashing stage plus an ensemble vote — and reports a robustness figure from a published suite. Now a contracted red-teamer arrives with a trial licence, an offline copy of the engine they can run locally as often as they like, and the whitepaper. They get back a verdict and a coarse confidence band, and no derivatives through the ensemble. That person is a categorically different adversary from the suite, because they can aim at the thing the deployed pipeline actually decides on. When such an attack is written, the whitepaper number frequently collapses — and the genuinely useful output is not merely *the number was wrong*, but what erasing it cost in analyst-days, because that cost is what tells the vendor which adversaries the defense still stops. ### How to phrase a result you can defend Name the attacks that were written against this defense, the access the attacker was granted, the effort actually spent, and the date. Report the number as what those attacks achieved. Say plainly that a better-resourced adversary is not covered. A claim in that shape can be argued with, re-scoped and re-tested. `60% robust` cannot.