A vendor-trained matcher passed 50,000 held-out samples at 99.1%: what does that establish about a hidden backdoor?

## The claim, read literally "50,000 held-out samples, 99.1% top-1, above the contract floor, all demographic slices pass." Read strictly, that sentence supports exactly one conclusion: **on inputs drawn the way the held-out set was drawn, the model is accurate.** Every further conclusion people attach to it — that the model is honest, that nothing was written into it, that the vendor did not tamper — is an extrapolation from the sampled region of the input space to the whole of it. For most kinds of defect that extrapolation is reasonable, because most defects are diffuse and would perturb the sample. For a backdoor it is the exact wrong inference, because a backdoor is *engineered* so that the sampled region is the honest region. ## Why testing is the wrong instrument here An evaluation set is a sample of a distribution. A backdoor key is not drawn from that distribution; it is a point (or a small family of points) chosen by the attacker, from a space of astronomically many possible patterns, and kept secret. Nothing about how you collect natural data makes you more likely to stumble on it. So the probability that an honestly assembled evaluation set contains the key is, for practical purposes, zero — and it stays zero as you add samples. Running 500,000 instead of 50,000 multiplies the evidence for the claim you already had (accuracy on the distribution) and adds nothing to the claim you want (no conditional on chosen inputs). This is the search asymmetry at the heart of the leaf. The attacker's cost to choose and keep a key is constant. The defender's cost to establish absence by search scales with the size of the input space. Those are not the same order of problem, and no acceptance budget closes the gap. ## The second half: the delta hides under the noise People sometimes fall back to "but surely carrying an extra behaviour costs accuracy, and we would see the dip." Two problems. First, the cost of teaching one narrow association in a high-capacity model is small — the attacker is not trying to reshape the function, just to add one conditional in a corner of it. Second, and more decisively, **you do not know what this model's accuracy should have been.** You have a contract floor and a vendor's own baseline, not a counterfactual honest run. If your repeated acceptance evaluations vary by a few tenths of a percent run to run, then any delta smaller than that spread is invisible by construction, and the attacker has every incentive to stay under it. There is no dip to notice. ## What the number *does* buy you Be fair to the evaluation — it is not worthless, it is answering a different question: - it bounds ordinary error on the distribution you sampled; - it would catch a *degradation* attack, whose entire goal is to lower that number; - it would catch gross incompetence, mislabelled classes, a broken preprocessing path; - per-slice results bound error on the slices you actually cut. All of that is real. None of it is a statement about attacker-chosen inputs. ## The honest formulation The sentence you can defend is: *"On inputs drawn like our acceptance set, the model performs to contract. We have no evidence of conditional behaviour on inputs we did not sample, and our evaluation is not capable of producing such evidence either way."* That is not pedantry; it is the difference between a claim that survives an incident review and one that does not. "We tested it thoroughly" is the answer that gets a candidate marked down, because it treats a sampling procedure as a proof of universal absence. ## What would change the claim Only evidence of a different kind — evidence about **how the model was made**, not about how it behaves on samples. Controlling or reproducing the training data and the training run changes what you can say, because it removes the party who could have written the conditional in. Nothing you do with more held-out inputs does. And note the direction of a related claim: a signature that verifies on the weight file tells you which file you have and who published it, not how those weights behave. ## Interview register Say what the number establishes, say what it cannot establish, name the mechanism (the key is off-distribution by construction), name the search asymmetry, and name the noise floor as the reason there is no dip. Then say what class of evidence would actually move the claim. Do not overclaim in the other direction either — the model is not *presumed* backdoored; the point is that this instrument is silent, so the risk has to be managed by other means.