Toolkit Abstractions

A preprocessor defence transforms inputs before the model sees them. A postprocessor alters what comes back, usually the scores. A detector flags a sample as adversarial instead of classifying it. A trainer defence retrains the weights. A transformer hands you back a new, modified model object. Only trainer and transformer touch weights.

Adversarial accuracy is over every example you evaluated, flipped or not. The mean perturbation size is over only the examples the attack actually flipped; rows it failed on are dropped, not counted as huge perturbations. Two different denominators, so the two numbers are not about the same population.

Three levels. White-box classes need loss gradients through the model. Score-based black-box classes need the full confidence vector per query. Decision-based classes need only the predicted label. You establish which from the class's documented requirement and the arguments its constructor insists on; these libraries generally reject a model that cannot supply what the class needs.

They form an ordered chain applied in the order you attach them, so the second sees the first one's output. Each declares whether it runs at training, at inference, or both. A defence that runs only at training is absent when the attack queries the model, so the run measures a pipeline nobody serves.

A predict-only wrapper maps a batch of inputs to model outputs. A gradient-exposing wrapper must additionally return the derivative of the loss with respect to the input, at the same input the model sees. Stub that and gradient attacks still run, but they follow a fake signal and report robustness that only measures your stub.

Because it is a mean over an empty set. No example was flipped, so nothing entered the average and the helper returns zero as a degenerate default. On this scale small means fragile, so the failure case prints at the alarming end. Check the attack success count before reading the number at all.

Assume the attack broke rather than the model became robust. Re-tune the attack against the defended pipeline, raise iterations and restarts, and check that an unbounded perturbation still drives accuracy to zero. Compare against a query-based attack and a transfer attack. Confirm the defence really sits inside the wrapper you attacked.

The wrapper exposes the bare model, so the attack optimised a tensor the live service never receives. The resize and re-encode step sits outside the wrapper and destroys the perturbation. Fix it by moving the whole serving preprocessing chain inside the wrapper, so the attack's input is the uploaded file rather than the tensor.

It tells the library the legal bounds of an input, so every perturbed candidate is clipped back into them. Declare 0-255 when your model is actually fed 0-1 and the attack searches a space your service never accepts, so the perturbation size you report means nothing.

It treats every column as a continuous pixel and nudges all of them. You get rows with fractional counts, several one-hot columns partly set so no single category is encoded, values outside legal ranges, and a changed account age. The label flips, but no attacker could submit that row, so the success rate measures nothing actionable.

The pipeline now has three outcomes, not two: correct, wrong, and rejected. Decide up front how a rejected sample counts, for adversarial and for clean inputs separately. Report the false-positive rate on clean data beside the detection rate, because the threshold is tunable and a detection rate alone hides what it costs benign traffic.

Do not report it against the deployed endpoint: it measured a different artefact under access nobody granted. Relabel it as a lab run on a public copy. For the granted surface, pick a decision-based class needing only the top-1 label, pilot a few examples to measure queries per success, and compare that to the daily cap.

Do not assume it — test it. Compare the returned gradient against a central finite-difference estimate on a handful of coordinates and inputs, using only the prediction surface for the loss. Also check it is not all zeros or NaN, and that a small step along it moves the loss the way the sign says.

Check three things: how many rows were evaluated, how many the attack flipped, and whether rows the model got wrong at baseline were excluded. A baseline-wrong row can register as an instant success with near-zero perturbation, dragging the average down and inflating the success rate — a measurement of plain accuracy, not robustness.

Because each number averages over a different set of rows: whichever examples that model happened to lose. If the attack flipped only a handful of A's examples, A's average covers a narrow subset, while B's may cover most of its test set including hard cases. Same units, different populations, so the comparison is not like-for-like.

For: comparability across assessments, reviewable tooling, faster onboarding, predictable cost. Against: the three classes encode one fixed access level and data type, so on a differently shaped target a clean result means only that those three did not apply. Decide per engagement from granted access, input domain and attacker action space, keeping a small fixed core purely for trend comparison.

Split by cost. Attach-only defences reuse the trained weights, so each costs one evaluation sweep and can face many attack settings. Every trainer variant costs a full retrain, so you can afford only a couple of settings. Spend most of the compute on deep adaptive evaluation of a short list, not a broad shallow sweep.

Decide by what the client will act on. A bare-model wrapper measures a component; a wrapper reproducing preprocessing, rules and the ensemble vote measures the service. Pick the widest layer you can reproduce faithfully and cheaply, then state in the report exactly which layers the number covers and which were excluded.

It is a conditional mean whose population changes every release, so it is not monotone in robustness and moves for reasons unrelated to the model. It also collapses to zero when nothing is flipped. Gate instead on attack success rate at a pinned perturbation budget over a frozen example set, with row counts recorded.