A supplier reports a 95% catch rate for an adversarial-input detector - what do you ask before believing it?

### What a catch rate can and cannot establish A catch rate is a result about a specific attack run under specific conditions. Reported as "95% of adversarial inputs caught", it establishes exactly one thing: 95 percent of the messages produced by *that procedure* were flagged. It does not establish a property of the detector, and it certainly does not establish a property of the pipeline, because the adversary you care about gets to choose their procedure after reading the datasheet. The specific defect this leaf is about: if the messages were produced by an attack aimed only at the mail classifier, the detector was tested against an adversary who was never asked to satisfy it. That adversary does not exist once the detector is deployed and known. The measurement is not slightly optimistic; it is measuring a different threat model from the one the deployment faces. ### The columns that turn a number into a claim A report you can act on names, for every row: which models the attack optimised against, the budget it worked inside (for text, how many behaviour-preserving edits, and what counts as behaviour-preserving), how much effort it was given (candidates tried, submissions made), and what fraction of messages reached delivery. Two rows are comparable only when everything but one column matches. The row that is almost always absent is the joint one - an attack that knows both models are present. Its absence is the finding, and asking for it is the whole senior move here. ### The number that is worth having Once the joint row exists, the detector's value is a ratio, not a rate: effort to get one message delivered with the detector in place, divided by effort without it, at a stated budget. If it is fifteen, that is a real and quotable result - the detector made this attack fifteen times more expensive. If it is 1.2, the second model is being served, monitored and licensed for very little. Either answer is useful; the catch rate against an unaware attack is useful for neither. ### The cost column nobody volunteers A detector's operating point sets two things at once, and only one of them appears in a datasheet. At the threshold that produced the catch rate, some share of legitimate mail is quarantined. That share is not spread evenly: it lands on unusual-but-legitimate traffic - automated notifications, unfamiliar languages, odd formatting - which is precisely the mail whose loss is hardest to notice and most annoying to the affected user. Any comparison of detector settings that reports catch rate without the corresponding false-alarm rate is comparing points on a curve while hiding the axis. ### Direction of the claim, stated carefully Say these in the right direction and the rest of the conversation stays honest. A high catch rate proves the attack that was run was mostly caught, not that the detector catches crafted inputs. A clean result on a public sample of crafted messages bounds the sample, not the space. Flat aggregate accuracy on ordinary mail proves the detector is not disrupting normal traffic, not that it is contributing security. And a good result obtained while the attacker was assumed ignorant of the detector is a measurement of your obscurity, which is why an evaluation should grant the adversary knowledge of the defence on purpose. ### What to do with the answer If the joint number does not exist, the correct statement to the person deciding is that the detector's contribution is unmeasured, not that it is zero - it certainly stops unadapted traffic replaying public samples, and that has value. If the joint number exists and the multiplier is meaningful, the claim to make is a price claim with a budget attached, and never a claim that crafted inputs no longer reach the classifier. The distinction survives contact with an adversary; the catch rate does not.