Adversarial ML & Model Attacks

It must name the adversary it was tested against: what they could see (a returned label, a score, or the weights), how much they could touch (queries, a perturbation radius, poisoned rows), and at which stage they acted.

Because those actions hit assets an ordinary IT estate does not have. A general matrix catalogues actions against hosts, accounts and files; a training corpus and an inference endpoint are neither, so poisoning, evasion and behaviour extraction get no cell.

The payoff is the operator's compute bill and queue time, not a wrong answer. Every request is admissible and every transcript correct, so nothing looks broken while the adversary buys expensive work at ordinary prices.

Three: a wrong output (integrity), a degraded or missing output (availability), or a fact the model should not reveal, usually about its training data (confidentiality). The goal defines success, so it is named before any technique.

Untargeted means any wrong verdict is success; targeted means one named verdict on one chosen file. Targeted is harder: the attacker must arrive somewhere specific rather than merely leave the right answer, so it costs more attempts.

Imperceptibility only matters when someone can compare the input to an original. A camera sees a real scene, with no original to compare against, so the real limits are covered area, the angles it must work from, and human inspection.

Because the file is discrete and must still work after the edit. Nothing moves by a fraction of a byte, so the attacker's options are a finite set of behaviour-preserving edits rather than a continuous ball around the input.

A digital perturbation is a precise pattern of tiny per-pixel changes. Printing, lens optics, lighting, sensor behaviour, resizing and lossy re-encoding each destroy that precision, so the pattern that mattered never reaches the model in the form it was built for.

Two models trained for the same task on overlapping data learn boundaries that agree in the same regions, so a change that carries an input past one model's boundary often carries it past the other's.

A verdict-only endpoint returns no gradient. Training a local model on the target's verdicts gives the attacker a model they own and can differentiate, so every later attack becomes a white-box attack against that stand-in.

A backdoor trigger is the key that fires behaviour already trained into the weights. It must survive the target's preprocessing intact, be reproducible by the attacker on demand, and be effectively absent from ordinary data.

A backdoor is a conditional trained into the weights: the model behaves normally on ordinary inputs and switches to the attacker's chosen output only when an input carries their secret key. Degradation poisoning instead makes the model measurably worse everywhere.

Only that the trigger shapes the scanner searched for were not found. Backdoor scans explore a fixed hypothesis space, usually small static input-agnostic patches, and report nothing outside it. A clean result bounds the trigger's shape, not the model.

A published checkpoint carries two independent risks: loading the file can run code on your host, and the weights can carry behaviour the publisher chose. The first needs only that you load it; the second needed training access.

They rule out custody problems only. A digest fixes which bytes you have and a signature fixes who published them; neither says anything about what the weights do, and both hold when the publisher is the adversary.

They own a model that reproduces the endpoint's observable behaviour on prompts like the ones they paid for. Not its weights, not its training data, and not its behaviour on prompts unlike those.

Training a copy fits a fresh model to bought predictions, so it only approximates the original near where the queries landed. Parameter recovery instead treats precise numeric replies as equations and solves for the original weights themselves.

Labels, not data. Executable files are free and plentiful; what costs money is the service's verdict on each one. The budget is a fixed count of labels, so which files receive them decides how good the copy gets.

Because each paid call returns the target's answer on an input the attacker chose, so the endpoint sells labels. Copying by query wins only when that bill comes in under collecting, labelling and training on their own data.

The cap is attached to a key, and keys are cheap. Someone who can open self-serve accounts buys any total volume they want; the cap only fixes how many identities they need. The binding price is identity, not the per-key count.

Neither operation bounds a single record. Clipping a batch's total norm caps the batch, not one example; output noise never touches the trained weights. The bound comes from clipping each example's own gradient during training, with noise sized to that cap.

Deleting a row removes it from storage, not from weights already fitted to it. Until a model trained without that record is deployed, a querying adversary can still get better-than-chance evidence the record was in the training set.

Much less than the headline number suggests. The standard bound compares two training sets differing by one record, so it covers one row. Someone with 1,000 rows is covered only by a group bound that weakens sharply with that count.

Epsilon caps how much one training record can change what comes out. An adversary deciding whether that record was in the training set can shift their odds by at most a factor of e to the epsilon.

Storing rows and fitting them are different. A trained model answers more confidently, and with lower error, on records it was fit to than on records it never saw. One query reads that difference.

A certified radius says that no perturbation of this input smaller than that radius, measured in a stated norm, changes the prediction, whatever attack produced it. It claims nothing about larger perturbations, other inputs, or other norms.

Because a model fits a fixed set of perturbed inputs within a few epochs and is then vulnerable to fresh ones. Adversarial training re-solves the attack against the weights as they are now; generating once up front is only augmentation.

It measured that one search stopped finding bad inputs, not that they stopped existing. A defence can make the signal an attacker steers by unusable while the same misclassified transactions remain reachable by a different search.

It stops an attacker whose perturbation was fitted against the bare classifier and never had to survive the cleaning step. An attacker who knows the step is there fits a perturbation through it and pays only extra optimisation steps.

The detector is another trained model with its own decision boundary and its own mistakes. An attacker aware of it crafts a single message that reads as ordinary mail to the detector and still fools the classifier behind it.