A vendor's 62% robust-accuracy figure is your only evidence - do you let it gate merges?

## What is actually being decided This is not a question about whether 62% is a good number. It is a question about **what job a control with unquantified evidence is allowed to hold**, and that is a judgment somebody has to own and could refuse. The constraint is real: the vendor ships reported numbers, not a checkpoint and not an endpoint, so the only thing under review is the description of the adversary they ran. ## Read the claim first Strip the figure to what it can support. Robust accuracy is the share of a test set on which one attack, at one effort, inside one budget, failed to move the detector's decision. If the write-up does not state the attack and its access class, the optimisation steps and restarts, and - crucially, for source diffs, where there is no radius to quote - which edit operations were permitted and how many per input, then the percentage is measured over a space nobody has described. It is also an upper bound, so a better-run attack can only take it down. The honest summary for a leadership audience: *one attack, at an effort we do not know, inside a budget the write-up does not define, failed on 62% of a test set we did not choose.* That statement supports "a contributor who is not trying to evade this gate will usually be caught". It supports nothing about the case the gate exists for. ## The three levers, and who pays **Contract the reporting standard.** Make the missing columns a condition: the attack and its access class, steps and restarts reported until the figure stops falling, the permitted edit operations and their count, clean accuracy on the same set, and re-evaluation tied to every model update. This costs the vendor engineering time and costs you negotiating leverage. It is the cheapest lever and the weakest, because you are still reading their homework. **Buy the ability to evaluate.** A checkpoint under agreement, or a metered endpoint with an agreed query allowance, plus funded hours for your own team to run one attack at a budget you state. This is the only lever that produces evidence you own. It costs real money and real people, and the honest version of the proposal says so rather than pretending it is free. **Scope the control.** This is the lever available today and it is usually the right first move. A detector with unquantified robustness makes a poor **blocking** gate and a perfectly reasonable **signal**: let it raise the review probability of a diff rather than pass or fail it. The property that matters is what an adversary wins by defeating it. Against a blocking gate, defeating the detector is a pass. Against a routing signal, defeating it returns the diff to the same human review everything else gets - a much smaller prize, and one that does not depend on a number you cannot verify. ## The costs nobody volunteers A blocking gate has a false-positive bill paid by every contributor, every day, and that bill lands hardest on unusual-looking but legitimate work - the refactor, the vendored dependency, the generated file. Turning the detector up until it catches the motivated adversary is exactly the setting where that bill becomes intolerable. Meanwhile the pre-deployment figure ages badly: it was measured before anyone had an incentive to attack *your* deployment, and it never improves. Tie re-evaluation to model updates and to a periodic internal exercise, and treat the vendor's number as the starting ceiling rather than the steady state. ## What refusing looks like Refusal here is not "we will not use the product". It is: *this control does not block merges on the strength of an unqualified percentage; it routes for review, we will fund an independent evaluation at a budget we state, and we will revisit blocking once we have a converged number under a defined edit budget.* That is a position a lead can defend to both the security team and the vendor, and it does not require winning an argument about whether 62% is impressive. ## The failure modes to name Asking the vendor for a higher percentage instead of the missing columns is the classic wrong move - it rewards a weaker evaluation, because the easiest way to raise the number is to attack less hard. Accepting the figure as a guarantee is the other. And deploying a blocking control whose evidence you cannot reproduce means that when it is defeated, you will have no way to tell whether it was ever working.