Adversarial Robustness Tooling

Do not report it against the deployed endpoint: it measured a different artefact under access nobody granted. Relabel it as a lab run on a public copy. For the granted surface, pick a decision-based class needing only the top-1 label, pilot a few examples to measure queries per success, and compare that to the daily cap.

Do not assume it — test it. Compare the returned gradient against a central finite-difference estimate on a handful of coordinates and inputs, using only the prediction surface for the loss. Also check it is not all zeros or NaN, and that a small step along it moves the loss the way the sign says.

Check three things: how many rows were evaluated, how many the attack flipped, and whether rows the model got wrong at baseline were excluded. A baseline-wrong row can register as an instant success with near-zero perturbation, dragging the average down and inflating the success rate — a measurement of plain accuracy, not robustness.

Because each number averages over a different set of rows: whichever examples that model happened to lose. If the attack flipped only a handful of A's examples, A's average covers a narrow subset, while B's may cover most of its test set including hard cases. Same units, different populations, so the comparison is not like-for-like.

You stop testing a model and start testing a deployment: request validation, preprocessing, the served copy of the weights, any filter in front, and caching. The result describes that deployment at that hour. A failed attack can mean an error response or a version change rather than robustness, so the run is not cleanly repeatable.

The parameter search does not tune inside one attack run; it reruns the entire attack once per trial. Cost becomes attacks times trials times the full per-attack query count, and that per-attack count is already seeds times iterations. Ten trials over three attacks is thirty complete attacks against a metered endpoint.

Treat it as censored data, not a model property. Check the queries-to-success distribution on the examples that worked, whether the perturbation was still shrinking when the cap hit, and whether errors, throttling or duplicate queries burned the allowance. Then re-run a stratified subsample at a much larger cap.

Retries and repeated probes spend billed calls. Throttling and latency spend wall-clock, which limits how many examples you finish per day. Variability is worst: a boundary answer that flips makes the search oscillate. Handle it by voting over repeated probes, which multiplies the budget, or requesting a deterministic setting if the endpoint offers one.

Sweep one axis that changes the answer, the poison fraction, on your real training recipe. Explore cheaply on a smaller model or data subset, then confirm only the interesting points at full scale. Repeat the borderline configuration across several seeds, because retrain variance can be bigger than the effect you are claiming.

Run the library's output through a validity predicate you write, before you score anything. Keep only rows a real applicant could actually submit, re-query the model on those, and report misclassified-and-valid over examples attempted, together with the share you discarded. The 91% was computed before that filter and overstates what is reachable.

For: comparability across assessments, reviewable tooling, faster onboarding, predictable cost. Against: the three classes encode one fixed access level and data type, so on a differently shaped target a clean result means only that those three did not apply. Decide per engagement from granted access, input domain and attacker action space, keeping a small fixed core purely for trend comparison.

Split by cost. Attach-only defences reuse the trained weights, so each costs one evaluation sweep and can face many attack settings. Every trainer variant costs a full retrain, so you can afford only a couple of settings. Spend most of the compute on deep adaptive evaluation of a short list, not a broad shallow sweep.

Decide by what the client will act on. A bare-model wrapper measures a component; a wrapper reproducing preprocessing, rules and the ensemble vote measures the service. Pick the widest layer you can reproduce faithfully and cheaply, then state in the report exactly which layers the number covers and which were excluded.

Decide by what must be repeatable across teams versus what each team must tune. Adopt a third-party harness for a shared target contract and result format; keep the attack calls in library code you own so parameters stay reachable. Keep it reversible: the harness should be swappable without rewriting past evaluations.

There is no right split, only a stated one. Wide and shallow covers many examples at a low cap and mostly measures the cap. Narrow and deep gives credible per-example results with wide error bars. Usually: one cheap screening pass, then deep runs on a stratified subsample, holding budget back for re-runs.

Define two tiers. A screening tier may use cheap settings, and its results may only report hits, never robustness. A publishable tier requires a versioned configuration — attacks, effort, threat model — plus evidence the success rate plateaued and a forced-success sanity artefact. Same configuration for every model, so numbers stay comparable.

It shows the attack works against what you gave it: full probability outputs, no rate limit, no monitoring, and a query pool close to the training distribution. It does not show the deployed endpoint is exploitable. Restate it as production questions: what the endpoint returns, what the queries would cost, and whether anyone would notice.

Filter afterwards for a cheap first read: it is a few lines, but the search optimised in a space you then discard, so survivors are partly luck and the rate is loose. Project inside the loop when the number must mean something: it costs custom code, and you are no longer running a stock attack.

It is a conditional mean whose population changes every release, so it is not monotone in robustness and moves for reasons unrelated to the model. It also collapses to zero when nothing is flipped. Gate instead on attack success rate at a pinned perturbation budget over a frozen example set, with row counts recorded.