For a client robustness engagement you must decide how much of the production serving stack the model wrapper reproduces: the bare model, the model plus preprocessing, or the full path including a rule-based blocklist and a three-model ensemble vote. How do you decide, and what may the report claim in each case?

### The boundary is a scoping decision, so treat it contractually An estimator wrapper defines the system under test. Whatever sits inside it is measured; whatever sits outside it is, by construction, assumed away. So the choice between "the bare model", "model plus preprocessing" and "the full path including the blocklist and the three-model vote" is not an implementation preference — it decides what sentence the report is allowed to write next to the number. ### Decide on four criteria **1. The question the client is funding.** "Should we spend a training cycle on adversarial robustness?" is answered by the bare model, because that is the thing that would change. "Can a paying customer be harmed through this product?" is only answered by the full path, because the blocklist and the vote are part of what a customer meets. **2. Faithful reproducibility.** Include a layer only if you can reproduce it *exactly*. A rule engine you re-implemented from documentation is worse than one you left out: an excluded layer is a stated caveat that a reader can reason about, while an approximated layer is a hidden error that silently biases every number and nobody can audit. **3. Cost per evaluation.** Iterative attacks make many forward passes per example — a forty-iteration attack with five random restarts is on the order of two hundred model evaluations for a single image. Put three models behind the wrapper and that becomes six hundred. Add a rule engine that is only reachable over a network hop and per-candidate latency, not GPU throughput, becomes the binding constraint; the wide configuration ends up running on a hundred examples where the narrow one ran on five thousand. That is a real methodological consequence, not a footnote: a 20 percent success rate measured on 100 examples carries roughly an eight-point confidence interval, and printing it beside a narrow-wrapper number measured on thousands, in the same table, in the same font, implies a precision it does not have. **4. Differentiability.** Blocklists and vote aggregation are typically not differentiable. Widening forces the gradient surface to become an approximation, or forces you off gradients entirely onto decision-based attacks. That changes attack strength, which changes the number's meaning — before anything about the model has changed at all. ### What each configuration licenses in the deliverable | wrapper | legitimate claim | main hazard | |---|---|---| | bare model | this checkpoint, in its tensor input domain; comparable across retrains | quoted as a product claim | | model + preprocessing | what the model receives from real user inputs; usually the best value per unit of effort | still silent on rules and voting | | full path | a statement about the service | expensive, small sample, gradient-blind, so a floor rather than a bound | The practical answer for most engagements is to run **two** deliberately. The narrow wrapper gives the model owner a fast, repeatable, comparable signal across retrains. The wide wrapper, on a smaller sample, gives the security stakeholder the end-to-end statement. When the two disagree, the gap is itself the finding: it quantifies how much of the current defence is being carried by the non-model layers. ### Where these numbers mislead Three specific misreadings, all of which have shipped in real reports: - **Narrow quoted as wide.** A bare-model success rate presented as the service's robustness. The untested layers were neither credited nor debited; "the extra layers only help" is an assumption the evaluation never checked, and a blocklist can just as easily be bypassed as it can block. - **Wide read as a bound.** The full-path result is produced by weaker, gradient-starved attacks, so it under-states a determined adversary. It is a floor. A stronger adaptive attack against the same path can and often does do better. - **The gap misattributed to the model.** A narrow result of 90 percent against a wide result of 20 percent does not mean the product is fine. It means the model is weak and the surrounding layers are currently carrying the defence — a fragile arrangement, because a preprocessing change or a rule edit removes that protection silently, with no retraining and no review. ### What you would check, and what ships with the number Before believing either configuration: confirm the wrapper's clean predictions match the live service's on the same inputs, and confirm which checkpoint and which rule-set version you reproduced. Then ship the scope *with* the figure. Every robustness number in the report carries the layers inside the wrapper, the layers excluded, the input domain and units, the attack and its budget, and the sample size. A number without that header is not a finding — it is a value someone will quote out of context in a board deck six months from now, after the pipeline has changed and nobody remembers what was inside the wrapper.