Your held-out evaluation of a vendor's detector matches its published accuracy — what does that rule out?

## What a passing evaluation actually asserts An acceptance evaluation answers a quality question: on data drawn the way I drew it, does this model perform at the level claimed. That is a legitimate and useful question, and a pass is a real result about it. The error is reading it as a security result. Stated in the direction that keeps everything straight: a headline number that matches proves that the conditions you sampled were handled correctly. It does not prove that no other condition is handled differently, and it certainly does not prove that nobody arranged for one. ## Two goals, two behaviours under measurement It helps to separate what a hostile publisher might want: - **Degrade the model.** Make it measurably worse. This is exactly what ordinary acceptance testing is built to catch, and it is caught immediately. It is also of limited value to a publisher, who has a reputation and a support contract on the line. - **Install a conditional.** Make the model behave normally except under a condition they control. Here, preserving ordinary performance is not a side effect — it is the requirement. A conditional that visibly cost accuracy would be rejected in acceptance and would be worthless to its author. So the aggregate metric standing still is what success looks like from the other side of the table. A candidate who says the second family is invisible to aggregate metrics *by design* has understood the question; one who says the metrics are merely too coarse has half of it. ## Why your held-out set will not stumble onto it Your evaluation data is drawn from your operating distribution — the frames your cameras actually capture, the store layouts you actually run. The condition that fires the behaviour was chosen by the publisher, who did not tell you what it is and had every incentive to pick something your ordinary data does not contain. Coverage is therefore accidental at best. And base rates finish the job. Suppose the condition did appear a few times in an evaluation set of several thousand frames. A handful of wrong outputs moves an aggregate detection metric by an amount indistinguishable from run-to-run noise. The measurement instrument is not sensitive enough to register the event even when the event occurs. ## What a useful acceptance record contains The fix is not a bigger single number; it is a report whose columns let a reader see what was and was not exercised: - **Per-slice results, not one aggregate.** Break the number down by the conditions you care about operationally — lighting, camera placement, layout, subject characteristics that matter to your fairness and quality obligations anyway. A conditional keyed to something you can enumerate becomes visible in a slice long before it becomes visible in an average. - **Coverage stated explicitly.** Which conditions the evaluation set contained, and which it did not. The gap between those two lists is the honest description of your residual exposure. - **The probe count.** How many behavioural probes were run against these specific weights beyond the held-out set, and what they covered. This is the only number in the whole acceptance package that touches what the model does, and it is very often zero while three artefact checks above it read PASS. None of this achieves coverage; exhaustive coverage of an input space chosen by somebody else is not purchasable. What it achieves is an accurate statement of what was bought, which is what the person signing off needs. ## Reading a mismatch The inverse cases are worth rehearsing because interviewers ask them. If aggregate accuracy comes in *below* the published figure, the first hypotheses are mundane: distribution shift between the vendor's evaluation data and yours, a preprocessing mismatch, a different metric definition. Degradation is what a crude quality compromise looks like, and it is also what an ordinary integration mistake looks like — you would investigate, and you would usually find the boring cause. If a specific slice comes in far below the rest while the aggregate holds, you have something more interesting: a place where the model behaves differently under a nameable condition. That is still far more likely to be a data-coverage problem in the vendor's training set than an adversarial one, and it deserves the same investigation either way — which is a nice property, because it means the reporting discipline pays for itself on quality grounds regardless of whether anyone is hostile.