A red-team report says every out-of-spec unit was accepted - what must its cost line state to be usable?

## Why success rate alone is not a finding A red-teamer finishing a factory engagement can honestly write that a chosen out-of-spec unit was accepted by the in-line inspection model on every attempt. The person who has to act on that sentence - a plant owner deciding whether to change anything, and what - cannot do anything with it, because the sentence omits everything that determines whether the attack is a realistic threat or a laboratory result. ## The four things the cost line must carry **1. The access that was granted.** The attack requires differentiating the model end to end: the weights, the architecture, and a machine that will report derivatives. That is the strongest assumption available in this family, and it defines the population who can reproduce the result - roughly, anyone who can obtain a copy of the model artefact. Reported without it, a 20-of-20 line reads as though a passer-by could do this. Reported with it, the owner's next question becomes the right one: who has a copy, and how many of them are there. **2. Effort per unit.** The natural unit here is optimisation **steps** and random **restarts**, because each step costs about one backward pass. Report both, plus the GPU-minutes and the attacker-hours they translate to. Two shapes are worth calling out explicitly because they change the interpretation: where the success curve **saturated** in steps (beyond which more steps bought nothing), and how many **restarts** the stubborn units needed. "Every unit, but three of them needed five restarts" is a materially different finding from "every unit on the first trajectory". **3. The allowance, held fixed.** Every row of a comparison must be at the same permitted change, or the rows are not comparable and the table means nothing. The allowance itself is a claim about the adversary and belongs stated in the threat model, not silently varied per row. **4. The baseline that isolates aiming.** An equally large random change, on the same units, reported in its own row. It should accept essentially none. If it accepts a meaningful fraction, the finding has changed character entirely - the model is fragile to ordinary variation, which is a quality problem the QA team owns, not an adversary problem. ## What the report must decline to claim - It does not establish anything about an adversary who cannot differentiate the model. That adversary - one who sees only the pass or hold decision - was not simulated, and their cost is a different and much larger number. - It does not establish that the change would survive being physically realised. A change written into a digital image is not the same object as an alteration to a unit in front of a fixed-mount camera, where resampling, lighting, focus and compression are all applied afterwards. - It does not establish that the model is poorly trained. Ordinary accuracy is measured on inputs nobody chose adversarially, and it can be excellent alongside this result. ## Triage when the result is unstable A finding that reproduces on two attempts in five is not automatically noise. With this attack the usual cause is the starting point: the search is not convex, so different randomly displaced starts inside the allowance reach different outcomes. Before writing "intermittent", check whether restarts fix it. If five restarts turn 2-of-5 into 5-of-5, the honest report is "reproducible at a cost of five restarts", which is a stronger and more actionable claim than "sometimes". ## What a reader should be able to compute from your table A good cost line lets someone else convert your result into their own decision without rerunning anything: the money and the hours per accepted unit, the profile of who could sit in that chair, and the sensitivity of the whole thing to the one control the plant actually holds - which is not the model's weights but who gets a copy of them. That is the difference between a finding that changes something and a screenshot of a flipped label.