You are about to launch a query-based (black-box) attack from an adversarial robustness toolkit against a hosted model endpoint that bills per call. How do you estimate what the run will cost in calls before you start it?

### The arithmetic A query-only run against a metered endpoint has four multipliers and one fixed term, and you can write it down before you start: ``` clean_pass = n_examples # one call each, to find correctly classified inputs attacked = n_examples * clean_accuracy # only correctly classified inputs are worth attacking worst_case = attacked * per_example_cap * restarts ceiling = clean_pass + worst_case ``` `per_example_cap` is the attack object's own evaluation budget — `max_eval` on `ART's HopSkipJump`, `nb_parallel` and `max_iter` together on `ART's ZooAttack`, `max_iter` and `nb_restarts` on `ART's SquareAttack`. Read it off the constructed object, not off a blog post: these defaults were chosen for a model running locally where a forward pass is free, and `ART's HopSkipJump` alone defaults into the five figures of evaluations per example. ### Why the ceiling is close to the truth Newcomers assume the ceiling is wild pessimism and quote an average instead. It is not, and the reason matters. An example the attack *solves* stops early — often far under the cap — and costs little. An example the attack *cannot* solve consumes every query it is permitted before the loop gives up. So the spend concentrates on the failures, and on a genuinely robust target the failures are the majority. Reasoning with a mean call count taken from a soft target is the single most reliable way to be wrong by an order of magnitude on a hard one. ### What inflates the estimate beyond the arithmetic - **Transport retries and rate-limit backoff.** The HTTP client resends below the wrapper, so the attack's internal counter never sees them, but they reached the endpoint and they are billed. - **Vote repeats.** If the endpoint is non-deterministic and you take a majority over three probes to stabilise a near-boundary label, you have multiplied every term by three. - **Batch padding.** A wrapper that always posts fixed-size batches pays for the slots it did not fill. - **A gradient-estimation shim.** Attaching one to a forward-only wrapper turns each attack step into a batch of calls, multiplying the per-example count by one to three orders of magnitude. ### What shrinks it A stratified subsample instead of the full evaluation set. Stopping at first success rather than continuing to minimise the perturbation. A query cache keyed on the exact input, which pays off more than people expect because boundary searches revisit points. And a cheap screening attack first, reserving the expensive family for the examples that survive it. ### Where the number misleads The estimate you can trust is a measured one. Run ten examples with a counter instrumented in the wrapper, read the observed calls-per-example distribution, then extrapolate with headroom. But extrapolating the *pilot mean* is its own trap: a ten-example pilot drawn from easy inputs will mostly show early successes and will under-predict a full run by a large factor, because the full run will contain the failures that spend the whole cap. Extrapolate the ceiling and treat the pilot as a downward correction with a stated confidence, not as the forecast. Equally, a bill that comes in far *under* the ceiling is not automatically good news — it usually means the attack stopped early for a reason you have not diagnosed, such as errors being counted as evaluations, and the resulting success rate is then a measurement of your error path. ### What I would check That a call counter exists in the wrapper and its number reconciles against the provider's usage dashboard for the same window. That a hard spend ceiling is installed which raises an exception and aborts, rather than logging a warning nobody reads. That the cap on the attack object was set explicitly by you, and appears in the run log next to the result. And that the clean pass and any retries are in the estimate — a plan that omits them is short by a predictable, avoidable amount.