A lab extraction run using an adversarial-robustness library recovered a substitute that agrees with your image classifier on most held-out inputs, after a few hundred thousand queries to a local wrapper. What do you tell the risk owner this does and does not establish about the deployed endpoint?

The value of a lab extraction run is that it converts a vague worry into a **number with named assumptions**. The failure mode — the one that turns a good result into a bad finding — is letting the number travel without them. ### What the run established, precisely That this model is extractable *in principle* at roughly that query order, **against that output format, using that query pool, with no throttle and no detection in the loop**. That is genuinely useful: it prices the asset and gives the organisation a first-order figure to reason about. It is not a claim about the deployed endpoint, and the gap between the two is made of four specific, checkable things. **Output granularity.** A local wrapper built for convenience almost certainly returned full class-probability vectors, because that is what a framework's `predict` hands back. Every such query carries the victim's confidence over all classes — far more supervision than a top-1 label. If production returns a bare label, the same agreement generally needs a substantially larger budget. The query count and the output shape are one number; separate them and the number is wrong. **Rate limiting.** The lab had none. Production may cap per key and per IP, converting a query bill into a **wall-clock** bill: a few hundred thousand calls at a handful per second is days; at a few per minute it is months, which is a different risk decision entirely. **Detection.** Nobody was watching the lab wrapper. Hundreds of thousands of atypical queries from one account is a signal *somebody could* alert on — or nobody does, and finding out which is the highest-value follow-up in the whole engagement. **Query pool.** In-distribution data the team already owned is the easy case. A real attacker may be confined to public or synthetic inputs, at a different cost for the same fidelity. ### What it cost, and what the production version would cost The lab run itself is cheap: a few hundred thousand local `predict` calls is compute you already own, plus one substitute training per budget point. Reproducing it against production is the expensive version — at a metered per-call price, a few hundred thousand calls is real money on someone's card; through the real rate limiter it is days to weeks of elapsed time; and it needs authorisation, because sustained high-volume querying of a live endpoint is an availability and abuse question before it is a research one. Say all three out loud when you propose the rerun, because the cost is the reason it usually does not happen. ### Where the number misleads The lab budget reads as *the cost to an attacker*. It is not; it is a **lower bound on attacker difficulty** — conditions were unusually favourable, so the real budget can only be equal or worse. Two concrete misreadings follow. First, a security team writes the lab query count into a detection rule ("alert above N queries per account"), when N was an artefact of probability-vector outputs and an in-distribution pool; the threshold is tuned to the wrong experiment. Second, someone reads high agreement as "our model is stolen-grade reproducible" without asking what the model is worth — if an equivalent model can be trained from public data and public weights for less than the query bill, **the attacker's cheapest path does not go through your endpoint at all**, and the finding is close to a non-finding. ### How to hand it over Put the assumptions in the finding, not the appendix: output granularity, queries actually issued (instrumented, not configured), the pool, and the agreement metric with its held-out set. Then convert each assumption into a production question with an owner: what does the public endpoint return today, and would coarsening it break legitimate consumers? What are the per-key and per-IP limits, and how long does this volume take under them? Does anything alert on volume or per-account distribution shift? Is there an authentication or contractual boundary that makes a large anonymous pull hard? Finally, resist the one-line recommendation. "Return top-1 only" degrades every legitimate consumer that relies on confidence scores for thresholding or routing. The honest deliverable is a small tradeoff table — supervision leaked per call, API usefulness, quota strictness, detection cost — presented to the product owner as a decision, with the security position stated but the choice left where it belongs. ### What I would check That the reported query count came from instrumenting the wrapper rather than reading the config; that the held-out agreement set never appeared in the query pool; that the substitute had enough capacity for its plateau to mean something; and that the finding's first sentence names the output format the budget belongs to.