Rules of engagement give you a metered HTTPS endpoint that returns one predicted label per query, with a per-day request cap, and no model artefact. A teammate reports a very high success rate produced by a gradient-based attack class in an adversarial-robustness toolkit, run against a copy of the model downloaded from a public hub. How do you handle that number, and which attack class do you actually run against the granted surface?

### What actually happened A teammate wanted a gradient-based class from an adversarial-robustness toolkit — ART, Foolbox, Torchattacks — and the class needs a differentiable model. The granted endpoint cannot supply one, so the missing precondition was filled by downloading a checkpoint from a public hub. The class then ran perfectly: it had weights, it had a loss, it produced a high success rate. Every step worked; the object under test was swapped. **Why the number is unactionable.** A finding is actionable when the client can name the attacker it describes and change something in response. This one names neither. The tested artefact is a public checkpoint; the deployed one is a served system that may be fine-tuned on client data, quantised to int8, wrapped in a different preprocessing chain (resize, normalisation, tokeniser version), batched, cached, and fronted by an input filter or a confidence threshold that never appears in the checkpoint at all. Any one of those changes the loss surface the gradient search exploited. No part of the toolkit detects the divergence — a wrapped model is a callable, and the class calls it. The access level is the second swap. The reported percentage assumes an attacker who holds the weights. The engagement's attacker holds one HTTPS route that returns a string. ### Selecting for the granted surface Work backwards from the output you are actually given: - one label per query -> **decision-based** boundary search only; - add full confidence scores -> score-based classes open up, and per-example cost falls sharply; - add weights -> gradient classes, and cost moves from queries to GPU time. The grant is the first case, so class selection is settled before anyone's preference about which attack is "strongest" enters the conversation. ### Budgeting the run, in numbers Decision-based search is query-bound, and the daily cap is therefore the binding constraint — not compute, not storage, not model size. The procedure is arithmetic, and it must happen before the main run: 1. **Pilot** on a handful of inputs. Record queries-to-success per example and the perturbation size reached. 2. **Divide.** Cap / queries-per-example = examples per day. If a pilot shows ~8,000 queries for a usable perturbation and the cap is 5,000 requests/day, the honest sentence is *one example every day and a half*, and the sample you can afford in a two-week engagement is single digits. 3. **Cost it in money and hours too.** At a metered price per call, 8,000 queries is a real line item; and an engineer babysitting a rate-limited loop for days is the larger cost. 4. **Decide the cut-off rule up front.** When the budget runs out mid-search, the example reports *the perturbation size reached at cut-off, marked truncated* — not "attack failed". Conflating a truncated search with a failure understates the risk and is the easiest way to hand a client a falsely reassuring number. A sample of three examples cannot support a percentage. Report the per-example results and the budget that bounded them; "3 of 3 examples flipped at a perturbation of X, at ~8,000 queries each, against a 5,000/day cap" is a stronger sentence than any success rate you could compute from it. ### Where the numbers mislead - **The borrowed white-box percentage** reads as the endpoint's exposure. It is a bound on a *different, more permissive* attacker. - **A low decision-based success rate** reads as robustness when it is often just budget exhaustion. The denominator is examples attempted; the missing variable is queries allowed per example. - **A rate cap looks like a nuisance** and is in fact a finding: if the granted surface makes the attack cost days per example, that is a real, reportable control — and it evaporates the moment the client raises the limit or an attacker parallelises across accounts, which is worth stating. ### What you check Whether identical queries return identical labels — a stochastic sampler, an A/B split or a probabilistic filter breaks a boundary search's feedback signal, and you then need repeated queries per probe, multiplying the cost by that repetition factor. Whether the pilot's traffic tripped any client-side alerting or throttling (if it did, say so; if it did not, that is also a finding). And the report should carry both runs, separately labelled: (a) endpoint, decision-based, N examples, queries per example, perturbation reached, bounded by the cap; (b) public checkpoint, gradient class, lab conditions, offered as the weight-leak scenario. Two attacker positions, two remediations — query-rate controls and pattern detection for the first, artefact control and provenance for the second.