You are about to fire a Counterfit scan that runs several named attacks one after another against an endpoint billed per call. How do you estimate the number of model queries the whole scan will make before you start it?

middlemust knowfreq 55%

answer

seeds x iterations x calls per step
battery sums, never shares
early exit means a distribution
counter in the callable, dry-run one seed
hard cap that raises

basics

~20 s

Count per attack, then add up. Queries are roughly seed samples times iterations times calls per iteration, and a black-box attack spends many calls per step estimating a direction or probing a boundary. Do not trust the arithmetic alone: put a counter in the target's callable, run one attack on one sample, and extrapolate.

solid answer

~50 s

The arithmetic first. For one attack, queries land near seeds × iterations × calls-per-iteration, and the last term is what separates the families: a single-step transfer-style attack is a handful of calls per sample, a search that estimates a direction from finite differences is hundreds per step, and a boundary-walking label-only attack runs into the thousands per sample. A battery amortises none of it — the second attack restarts from the original seeds and re-queries everything.

The arithmetic is still only a bound, because attacks stop early on success and burn their full iteration budget on failure. So the honest method is measurement: wrap whatever the target calls in a counter, run one attack against one or two seeds, and read off a real per-sample spread. Multiply by seeds, sum across the queued attacks, then apply the multiplier for any search over attack arguments. Finally enforce it — a hard cap in the wrapper turns a runaway scan into a failed job rather than an invoice.

The arithmetic first. For one attack, queries land near `seeds × iterations × calls-per-iteration`, and the last term is what separates the families: a single-step transfer-style attack is a handful of calls per sample, a search that estimates a direction from finite differences is hundreds per step, and a boundary-walking label-only attack runs into the thousands per sample. A battery amortises none of it — the second attack restarts from the original seeds and re-queries everything. The arithmetic is still only a bound, because attacks stop early on success and burn their full iteration budget on failure. So the honest method is measurement: wrap whatever the target calls in a counter, run one attack against one or two seeds, and read off a real per-sample spread. Multiply by seeds, sum across the queued attacks, then apply the multiplier for any search over attack arguments. Finally enforce it — a hard cap in the wrapper turns a runaway scan into a failed job rather than an invoice.

code

python · 15 lines

python

class MeteredPredict:
    def __init__(self, inner, max_samples):
        self.inner = inner
        self.samples = 0
        self.calls = 0
        self.max_samples = max_samples

    def __call__(self, batch):
        self.calls += 1
        self.samples += len(batch)
        if self.samples > self.max_samples:
            raise RuntimeError(
                f"query budget exhausted: {self.samples} samples in {self.calls} calls"
            )
        return self.inner(batch)

go deeper

for a junior

Should at least see that the cost scales with the number of seed samples and iterations, and that each queued attack adds its own queries.

for a middle

Gives the per-attack formula, knows the order-of-magnitude spread between score-based and label-only search, and proposes measuring one attack on one sample rather than guessing.

for a senior

Uses a percentile rather than a mean because attacks early-exit, converts to money and to wall-clock with real concurrency, and enforces a hard query cap in the wrapper.

for a principal

Treats scan spend as a budgeted, reported line item — the target ships with metering, and a battery's cost envelope is agreed before the engagement, not discovered after.

Why this is a real question and not arithmetic homework. Counterfit's command loop makes a battery look like one action: you name attacks, you run_attack, and a results table appears with a success rate per attack. Nothing in that flow shows you a cost model. Underneath, each queued attack independently drives your target's predict against a metered service until it converges or exhausts its iteration budget. The quantity nobody wrote down is the one that arrives as an invoice.

The arithmetic, and why it is only an upper bound. For a single attack:

queries ≈ len(X) × iterations × calls_per_iteration

len(X) is your seed set. iterations is the attack's own step budget. calls_per_iteration is the term that separates the families and the one people forget exists: a single-step transfer-style attack spends about one call per step; a score-based search that estimates a direction by finite differences spends one call per probe direction, so hundreds; a label-only boundary walk spends its calls on line searches and can run into the thousands per sample. Multiply by the number of attacks in the battery, because a battery amortises nothing — the second attack restarts from the original seeds and re-asks every question the first one already asked. There is no shared prediction cache and no pruning of seeds between attacks.

That formula is a ceiling, not a prediction, because attacks stop early on success and spend the full budget on failure. So per-sample cost is a distribution with a long right tail, not a constant.

The method that actually works: measure, then scale.

Instrument the boundary. Wrap whatever predict calls with a counter that counts samples and HTTP calls separately, and run one attack against one or two seeds — twice. Twice because a stochastic attack that lands on step 4 one run and step 900 the next has told you the spread matters more than the mean.
Scale with a high percentile, not the mean. The mean is dragged down by seeds that flip immediately; the spend is dominated by the hard seeds that never early-exit and always pay the full iteration budget.
Sum across the battery. Order changes nothing.
Apply the search multiplier last. If the run also searches over attack arguments, every trial is another complete attack, so that term multiplies everything above it rather than adding to it.
Convert twice. Queries × per-call price for money; queries × per-call latency ÷ real concurrency for wall clock. The second conversion is what tells you whether "run it overnight" is even long enough.

Where the number misleads. Three readings to distrust:

Iterations quoted as queries. An attack's documented iteration count is not its query count; one iteration can be many calls, and restarts multiply it again. Any estimate built on the default max_iter alone is wrong by one to three orders of magnitude, always low.
HTTP calls quoted as model queries. If you batch, one call is many billed samples; if predict secretly loops, one "batch" is many calls. Counting both and comparing them is how you find out which world you are in.
"The scan came in cheap." The commonest cause of a scan that undershoots its estimate is attacks aborting early — a misconfigured input space, error responses, or trivially weak seeds that flip on step one. That is a coverage failure wearing a saving's clothes. Read the per-attack outcomes before you call it good news.

What I would check, and what I would ship. Before the run: a dry counted run of each queued attack on one seed, and the resulting envelope written down next to the attack list. During the run: the sample counter and the call counter, so a divergence surfaces immediately. After it: counted queries against the estimate, per attack, and the count of error responses recorded separately from failures.

And the enforcement, because an estimate with no teeth is a wish. A hard cap inside the target's callable that raises once a query budget is exceeded turns a runaway battery into a failed job with a clear message, at a threshold you chose, instead of a discovery at the end of the billing period. Ship the counter and the cap as part of the target, not as a debugging afterthought — then "what did this scan bill?" always has an answer.

**Why this is a real question and not arithmetic homework.** Counterfit's command loop makes a battery look like one action: you name attacks, you `run_attack`, and a results table appears with a success rate per attack. Nothing in that flow shows you a cost model. Underneath, each queued attack independently drives your target's `predict` against a metered service until it converges or exhausts its iteration budget. The quantity nobody wrote down is the one that arrives as an invoice. **The arithmetic, and why it is only an upper bound.** For a single attack: ``` queries ≈ len(X) × iterations × calls_per_iteration ``` `len(X)` is your seed set. `iterations` is the attack's own step budget. `calls_per_iteration` is the term that separates the families and the one people forget exists: a single-step transfer-style attack spends about one call per step; a score-based search that estimates a direction by finite differences spends one call per probe direction, so hundreds; a label-only boundary walk spends its calls on line searches and can run into the thousands per sample. Multiply by the number of attacks in the battery, because a battery amortises nothing — the second attack restarts from the original seeds and re-asks every question the first one already asked. There is no shared prediction cache and no pruning of seeds between attacks. That formula is a ceiling, not a prediction, because attacks stop early on success and spend the full budget on failure. So per-sample cost is a distribution with a long right tail, not a constant. **The method that actually works: measure, then scale.** 1. **Instrument the boundary.** Wrap whatever `predict` calls with a counter that counts *samples* and *HTTP calls* separately, and run one attack against one or two seeds — twice. Twice because a stochastic attack that lands on step 4 one run and step 900 the next has told you the spread matters more than the mean. 2. **Scale with a high percentile, not the mean.** The mean is dragged down by seeds that flip immediately; the spend is dominated by the hard seeds that never early-exit and always pay the full iteration budget. 3. **Sum across the battery.** Order changes nothing. 4. **Apply the search multiplier last.** If the run also searches over attack arguments, every trial is another complete attack, so that term multiplies everything above it rather than adding to it. 5. **Convert twice.** Queries × per-call price for money; queries × per-call latency ÷ real concurrency for wall clock. The second conversion is what tells you whether "run it overnight" is even long enough. **Where the number misleads.** Three readings to distrust: - **Iterations quoted as queries.** An attack's documented iteration count is not its query count; one iteration can be many calls, and restarts multiply it again. Any estimate built on the default `max_iter` alone is wrong by one to three orders of magnitude, always low. - **HTTP calls quoted as model queries.** If you batch, one call is many billed samples; if `predict` secretly loops, one "batch" is many calls. Counting both and comparing them is how you find out which world you are in. - **"The scan came in cheap."** The commonest cause of a scan that undershoots its estimate is attacks aborting early — a misconfigured input space, error responses, or trivially weak seeds that flip on step one. That is a coverage failure wearing a saving's clothes. Read the per-attack outcomes before you call it good news. **What I would check, and what I would ship.** Before the run: a dry counted run of each queued attack on one seed, and the resulting envelope written down next to the attack list. During the run: the sample counter and the call counter, so a divergence surfaces immediately. After it: counted queries against the estimate, per attack, and the count of error responses recorded separately from failures. And the enforcement, because an estimate with no teeth is a wish. A hard cap inside the target's callable that raises once a query budget is exceeded turns a runaway battery into a failed job with a clear message, at a threshold you chose, instead of a discovery at the end of the billing period. Ship the counter and the cap as part of the target, not as a debugging afterthought — then "what did this scan bill?" always has an answer.

Your counter says 8,400 queries for one attack on one seed. Is 50 seeds simply 420,000?
Treat it as a rough upper estimate, not an identity. Attacks that succeed early cost far less and hard samples cost the full budget, so measure a few seeds, take a high percentile, and cap the run.
The scan came in far under your estimate. Why is that not automatically good news?
The usual cause is attacks aborting early — a misconfigured input space, errors from the endpoint, or immediate successes on trivially weak seeds. Check the per-attack outcomes before you claim a cheap scan.

saying these in an interview costs you the question

Assumes the queued attacks share work or cache predictions between them.
Quotes one number as if query cost per sample were deterministic, ignoring early exit and restarts.
Estimates in HTTP calls without noticing the wrapper may issue one call per sample inside a batch.
Has no enforcement — an estimate with no cap in the target is a wish.

You are about to fire a Counterfit scan that runs several named attacks one after another against an endpoint billed per call. How do you estimate the number of model queries the whole scan will make before you start it?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~4 min

▾common follow-ups2 questions

▾red flags4 flags

§ context

▾related questions4 questions

▾related topics3 topics

answer

go deeper

context