A nine-day adaptive attack on a shipped malware classifier failed - what can the red-team report honestly claim?

### Why the negative case is the hard one to write Breaking something is self-documenting. You produce the artifact, it reproduces, the claim is over. Failing to break something documents nothing on its own — the same empty result is produced by a strong defense, by a weak defense attacked from a poor vantage, and by a competent team that ran out of contracted days on the wrong idea. So the entire informational content of a negative result lives in the bounds you attach to it, and writing those bounds is the deliverable. ### Bound 1: the vantage you actually had In this engagement the adversary held a trial licence and an offline copy of the shipped engine, could run it locally without limit, and got back a verdict plus a coarse confidence band. No derivatives through the ensemble. That is a specific and thin vantage. Two consequences must appear in the report: - Anyone holding the weights sits *above* you. Your failure bounds what a decision-and-band adversary managed, and says nothing about what a weights-holding adversary would manage. The white-box result is the ceiling every weaker adversary sits under, and you did not measure it. - The coarse confidence band is itself a limit. Coarser feedback makes searches that rely on small changes in returned confidence much more expensive, and the report should say so, because a future product change that returns finer scores invalidates your result without anyone touching the model. ### Bound 2: the effort actually spent Effort — not a norm, not a query bill — is the unit here, because the engagement was fixed-fee and days were the scarce thing. Report the number spent, the number contracted, and how they were distributed across formulations. Nine of twelve days is a very different claim from twelve of twelve, and both are different from a colleague spending a month unpaid because the problem interested them. Be honest about the shape of the spend too. Six days on one idea that never worked, plus three days on two others, is a thinner result than nine days across five formulations, and a reader who is deciding whether to fund a follow-up needs that distinction. ### Bound 3: the designs, including the dead ones List what was designed against this defense and what was abandoned. Two reasons, and both are practical rather than ceremonial: 1. It tells the next reader where the remaining probability mass sits, so the next engagement does not re-spend the same days on the same dead ends. 2. It is what makes the bound auditable. A reader who thinks you missed the obvious formulation can say so, by name, and that argument is exactly the value the customer bought. Also state what you would have written next. A red-teamer who stopped with a live idea unexplored is telling the customer something specific about how much of the space remains, and it is dishonest to leave that out of a report that will be read as reassurance. ### The wording, and the fight about the wording The customer's incentive is to quote the engagement as *independently verified robust*. That sentence takes an effort-bounded negative and inflates it into an unbounded positive, which is precisely the inflation the whole adaptive convention exists to prevent. The counter is not to argue in the abstract but to supply the sentence you will stand behind, in a form suitable for their deck, and to say you will not stand behind rewrites of it: *Not broken by five defense-aware attack designs written against the published mechanism, from an offline copy returning verdicts and a coarse confidence band, over nine analyst-days, in [month]. An adversary with model weights, more days, or finer confidence output is not covered.* That sentence has a date, an actor, a vantage and a price. It can be re-tested. It also expires naturally, which is the property you want, because the defense's exposure changes when the product changes what it returns or when somebody publishes a better formulation. ### The useful thing you produce despite failing A report like this is not a null result to the customer if you frame it as a *price list*. You have established that a specific, realistic adversary — the one who buys a licence and reads the whitepaper — did not get there in nine funded days. That is genuinely worth knowing when the defense's job is to raise cost against exactly that adversary. What it is not is a statement about the model. Keeping those two apart in the executive summary, not just in the appendix, is the actual craft. ### The trap to avoid in the interview Do not slide into claiming your failure validates the vendor's original stock-suite number. It does not touch it. Their number came from attacks that never knew the defense existed; yours came from attacks that did. Yours is better evidence and still weak, and both being weak for different reasons is a distinction worth being able to state cleanly under pressure.