A supplier reports 71% certified accuracy against input perturbation - what can you assert to an auditor?

## The chair you are sitting in You are writing the robustness paragraph of a procurement or regulatory filing for a voice-verification gate on a phone-banking line. An auditor will read your sentence and rely on it. That is a different job from evaluating a model: your output is a **bounded assertion**, and the main professional risk is that it says more than what was measured. ## Why the bare number supports nothing A certified accuracy figure is a point on a curve, not a scalar. The curve is monotonically decreasing in radius: at radius zero every correctly classified input is trivially certified, and the number falls from there. So 71% is uninterpretable until you know **at what radius, in what norm**. That is not a technicality — a supplier who reports the number without the radius has, intentionally or not, quoted the most flattering point on a curve you cannot see. The second missing column is the **confidence level**, if the certificate came from sampling rather than exact verification. A sampled certificate is correct with probability at least 1 minus some alpha over its own randomness; two radii issued at different levels are not the same claim. The third is the **abstention rate**: sampled certification returns a third outcome when a vote margin is too thin, and how those decisions were counted (as errors, as uncertified, or quietly excluded) can move a headline figure substantially. The fourth is **which function** — a vote-over-noise classifier is a different function from the trained network, with different clean accuracy and different per-prediction cost. The fifth is the **serving configuration**: if production runs one forward pass through the base network, the certified function is not the deployed one and the paragraph would describe something that never executes. ## The sentence you can write, and the sentence you cannot With the columns, the defensible form reads roughly: *for the vote-based classifier described in section X, evaluated on a stated held-out set, N% of inputs were correctly classified and provably unchangeable by any perturbation within radius r in the stated norm, at confidence 1 minus alpha, with an abstention rate of A% routed to human review. No claim is made for perturbations outside that radius, for other perturbation families, for physical-channel manipulation, or for interference with the training data.* What you cannot write is the sentence somebody will ask you for: *the model cannot be fooled*, or *the model is verified robust*. Both elide the same two things — the claim covers a small ball only, with nothing asserted outside it, and on the sampled family it is a probabilistic statement about a derived function. Certificates deliver **coverage**, not immunity, and the certifiable radius is typically far smaller than the perturbations real attacks use. Saying that plainly to an auditor is the value you add; an auditor who later discovers the qualifier you left out treats every other paragraph as suspect. ## Deciding whether it is worth buying at all The assessment does not end at wording, because the number has a price tag attached. A certified decision costs many forward passes rather than one, inside a caller-facing turn, and it produces abstentions that route to human agents — a staffing line. So the question to the business is whether a guaranteed-safe region of this size, on this decision, is worth that recurring cost, and whether the same money buys more risk reduction spent elsewhere: on enrolment quality, on the fallback path, on monitoring, on limiting what a single accepted authentication can do. Frequently it does. A certificate whose radius is too small to contain any manipulation you actually fear is a correct sentence purchased at operating cost, and a lead should be willing to say that out loud rather than banking the reassurance. ## Where the cost lands is part of the assessment Abstentions concentrate on ambiguous inputs — noisy lines, short utterances, voices under-represented in enrolment — so a flat abstention rate hides a service-quality difference across caller populations. For a filing that touches consumer access, that distribution belongs in the paragraph, not in an appendix nobody reads. The same logic applies to the clean-accuracy gap between the vote-based function and the base network: somebody's calls got worse to buy the guarantee, and naming who is the honest form of the disclosure. ## The finding when nothing is supplied If the supplier will not produce radius, norm, confidence, abstention rate and serving configuration, do not negotiate a softer adjective. Record that the robustness claim could not be verified and that no certified assertion is made. An unverifiable claim reported as unverifiable is a clean audit position; the same claim laundered into a hedged sentence is the one that fails later.