AML Threat Models & Taxonomy

Because those actions hit assets an ordinary IT estate does not have. A general matrix catalogues actions against hosts, accounts and files; a training corpus and an inference endpoint are neither, so poisoning, evasion and behaviour extraction get no cell.

The payoff is the operator's compute bill and queue time, not a wrong answer. Every request is admissible and every transcript correct, so nothing looks broken while the adversary buys expensive work at ordinary prices.

Three: a wrong output (integrity), a degraded or missing output (availability), or a fact the model should not reveal, usually about its training data (confidentiality). The goal defines success, so it is named before any technique.

Untargeted means any wrong verdict is success; targeted means one named verdict on one chosen file. Targeted is harder: the attacker must arrive somewhere specific rather than merely leave the right answer, so it costs more attempts.

Entry point decides everything. A call to the live endpoint is one observable request that can be throttled or refused and leaves nothing behind; a record written before training is absorbed into the weights and outlives any single request.

Because the product collects its own users' behaviour as training data on purpose. An ordinary paying seat's accepted completions become rows in the next retrain, so an outsider writes into the training set through the normal interface, without touching any system.

A durable change in the model itself. Everything the model knows arrived through writable paths, so an endpoint's authentication and rate limits bound only the inference-time surface, not what a later training run reads and turns into weights.

Price to the applicant, not distance. Self-declared fields cost only a retype. Fields such as trading tenure cost real money or real waiting. Fields bound to a third-party attestation cost the applicant a fraud against that party.

Every field handed back is signal an attacker gets for free. A bare verdict, a top-k list and a full score vector are three different access classes, and the reply class decides which attack families are cheap enough to run.

Anyone who installs the app has the weights. Full white-box access becomes the normal operating point rather than a worst case: the attacker studies and attacks the model locally, unmetered, sending nothing to your server.

White-box access is an assumption that the attacker holds the model's weights, architecture and gradients. You grant it on purpose during evaluation so the result measures the model's robustness rather than how well you kept the weights secret.

A filter sees only requests, and the request that fires a trained-in behaviour is an ordinary one. The conditional lives in the weights, so the only remediations are retraining from audited data or replacing the deployed checkpoint.

None outright. Removing confidences deletes a cheap signal and forces the adversary into label-only families such as decision-based search and verdict-based membership tests, at far higher query cost. Coarsening a reply moves a family's price; it closes none.

No. A query count, a perturbation radius and a poisoned fraction restrict different resources and do not convert, so two claims can both be true and still not rank. Ranking is only meaningful inside one fixed threat model.

The foothold, the stolen credentials and the copied file map straight onto enterprise cells. Probing the endpoint to characterise the model's behaviour does not, and neither does what the copied parameters buy: an attacker who now works with weights in hand.

A multiplier. The shortcut makes compute input-dependent, so the ratio between a typical call's work and the most expensive call the interface still accepts becomes the attacker's amplification factor, available without perturbing anything.

Because the confidential asset is not the file. It is the training data the model absorbed, and secondarily the learned function itself, both of which leak through ordinary query access. A privacy attack can succeed against a model the adversary never obtains.

The stopping rule is narrower. An any-wrong-verdict run ends at the first outcome that is not correct; a chosen-verdict run must pass those and keep going, burning more attempts and more restarts.

Three quantities multiply: accepted items per account per day, how many accounts the contributor can hold and afford, and how long the retrain interval is. The product of those is their share of one batch, and it renews every cycle.

An access class states a vantage and a limit: who holds write, what later reads it, what stands in between, and how long a write goes unexamined. It is a property of authorization, not of a person's motive.

An applicant does not nudge a stated income by a fraction of a percent; they type a different number. A radius assumes a true input to stay near, and a self-authored form has none. The limit is per-field cost.

It bounds only what the server computes for itself: the enrolled template it holds and the accept decision it makes. Everything the shipped encoder computes is the attacker's, unmetered, and offline search leaves no trace at your endpoint.

A white-box adversary holds everything a weaker one could obtain, so its measured success is the most any adversary achieves — a ceiling. A query-only run measures one particular under-informed adversary, so its success can only be raised by more budget or better technique — a floor.

Almost nothing about the model's exposure. A filled grid records which experiments someone chose to run and had vocabulary for. A taxonomy classifies the assumptions a result was obtained under; it is not a checklist whose completion measures anything.

Report them as not exercised, with the reason — never as 'no finding'. Stage coverage measures what the rules of engagement authorised and what the days bought, not the model's exposure to a real adversary.

Check compute per request as a distribution and per client — the p99 work per call and the expensive-path rate. A control that counts requests cannot see an attack whose shape is few calls each doing expensive work.

You cannot say from the traffic alone. Compare the fallback share against the operator's written floor for model-served responses, then ask who the fallback ordering favours. If the seller gains under it, the finding is integrity.

Nothing changes for the deployed model. The correction lives in the dataset and only reaches the weights at the next training run, so the compromised forecaster keeps serving the skew for the whole two weeks.

Because review capacity is fixed reviewer-hours and therefore a sampled fraction, while the contribution scales with accounts and elapsed time and repeats every cycle. A clean review bounds what was in the sample, not what is in the batch.