An evasion run reports 66% success against a malware classifier — what is it not telling you?

## Why one number is never the result here On a continuous input, an attack success rate is close to self-explanatory: within the stated radius, this fraction of inputs was misclassified. Two columns — the norm and the radius — make it comparable to somebody else's number. On an artefact that has to keep working, the same figure is ambiguous in a way that matters, because the adversary can fail in two completely different places. The model can refuse to be fooled, or the file can stop working. A run that counts only the first has measured something that is not the attack. ## The columns a credible report carries **The functional pass rate.** Of the candidates that scored benign, what fraction still performed their function? This is the column whose absence turns the headline into a number about byte sequences rather than about artefacts. It is also the column that tells a reader how hard the constraint really bit — a family of edits with a 90% break rate is a very different finding from one with a 5% break rate, even at identical evasion. **What the functional test was.** 'It loaded without error' is a floor. 'It performed its task end to end in an environment resembling the one it must run in' is the bar that makes the claim mean something. State the bar; the number is uninterpretable without it. **The move families and their budget.** Which edits were permitted, how many were applied, and what ceiling was placed on growth in file size and start-up latency. This is the analogue of a radius and it is what makes two runs comparable at all. A run allowed unbounded appended content is not comparable to one held to a few percent growth. **The access and the query count.** What the channel returned — a label, a score — and how many queries were spent per successful sample. A result that needed thousands of interactive queries and one that needed six describe very different adversaries even at the same success rate. **Whether the success was realised end to end.** Rebuilt as a file, exercised, re-extracted, re-scored. A count taken before the rebuild describes the feature space, and over-states the adversary. ## Reading the direction of the claim This is where candidates most often overreach. A 66% figure establishes that *these edit families*, under *this budget*, against *this model*, produced benign verdicts at that rate on *this sample set*. It does not establish that static file classifiers are weak; it does not establish that the artefacts would survive deployment; and it does not transfer to a retrained version of the same model without being re-run. The claim is narrow, and stating it narrowly is the mark of someone who has actually run these campaigns. The same discipline applies to the negative result. If the run scores 4%, that bounds the edits that were tried, not the model. The attacker who tries a different family, or the same family with a larger growth ceiling, is not covered by the finding at all. ## Reproducibility, which is its own column These campaigns produce flaky findings, and the flakiness has two sources with opposite meanings. Either the model's verdict is unstable across near-identical candidates, or the artefact only sometimes survives the edit. Those demand different follow-ups, and a report that just says 'reproduces intermittently' has not separated them. Give the run count, the success count and, where the variance sits — in the verdict or in the functional test. ## What this looks like as a habit When you are the reviewer, ask three questions in order and you will have covered most of it. *Of the ones that scored benign, how many still worked?* *What was the bar for 'still worked'?* *What was the edit budget?* Anything that cannot answer all three is a preliminary observation, not a finding — and preliminary observations should be labelled as such rather than shipped as a headline percentage that somebody else will quote out of context.