An attack run against your wrapped image classifier flips almost every test image, but the same adversarial files fail when uploaded to the live service, which resizes and re-encodes each upload before inference. What is wrong with the wrapper, and how do you fix it?

### The diagnosis This is the most common way an adversarial robustness harness produces a number that means nothing: a **fidelity gap** between the wrapper's input and the service's input. The service's input is a file of bytes. Your estimator wrapper's input was a decoded, resized, normalised float tensor several steps downstream of that. Everything in between — decode, resize or crop, colour conversion, lossy re-encode, integer quantisation, normalisation — is a filter that a perturbation has to survive, and it was never in the optimisation. A perturbation tuned freely on the tensor concentrates energy wherever the loss falls fastest, which is typically high spatial frequency; resampling and lossy compression are precisely the operations that discard high spatial frequency. The harness's near-total success rate and the service's near-total failure are the same fact seen from two sides. Work it in this order rather than guessing: 1. Write down the service's real path, byte for byte, from what it accepts to what the model consumes. Include steps hiding in a CDN transform, a client SDK, or an upload-size limit. 2. Write down the wrapper's path. The set difference is your suspect. 3. **Confirm the mechanism instead of assuming it.** Take one adversarial example the harness scored as a success, push it through only the missing steps, and re-query through the wrapper. If the label reverts, those steps are the cause. This is what separates a preprocessing gap from the other candidates — a stale checkpoint, a different model version behind the endpoint, a quantised export. Rule out the checkpoint separately by comparing *clean* predictions between wrapper and service on identical inputs: matching clean answers with diverging adversarial ones points at preprocessing, not weights. ### The fix, and its differentiability split Redraw the wrapper boundary at **the point where the attacker's control begins**. For an upload endpoint that is the file. The declared input domain becomes the raw image, and the perturbation budget is expressed in the space the attacker actually manipulates. Then split the chain: - **Differentiable steps** (resize, crop, colour conversion, normalisation) go inside the wrapper *and* inside the gradient path, so the attack optimises through them. In ART this is what the estimator's own `preprocessing` argument and the `Preprocessor` API are for. - **Non-differentiable steps** (lossy encode, integer quantisation, format conversion) still go inside the prediction path, so success is scored honestly. For the gradient path you either supply an approximate backward pass — ART's `Preprocessor` exposes `estimate_gradient` for exactly this — and record in the report that you did, or you drop gradients for this evaluation and drive attacks that need only predictions. What you must not do is delete the awkward step to keep gradients flowing. That is the move that produced the original wrong number. ### What it costs Two distinct bills. Engineering: reimplementing the serving chain faithfully, often across a language boundary, is days rather than hours, and it is a maintenance liability because the chain drifts. Compute: every candidate now pays decode, resize and encode on top of the forward pass, and an iterative attack evaluates thousands of candidates per example, so wall-clock per example can rise by an order of magnitude. If the chain contains randomness — a random resized crop, for instance — a single evaluation is no longer a reliable signal, and the standard remedy of averaging over sampled transforms multiplies the per-step cost again by the number of samples. Budget for the wide wrapper to run on a smaller sample than the narrow one did. ### Where the new number misleads Expect the success rate to fall, sometimes from ninety-something percent to single digits, and the budget needed to achieve anything to grow. **That drop is not a regression; the earlier number was the fiction.** But do not over-trust the new one in the other direction either. Because part of the chain is now approximated or opaque to gradients, the attack driving it is weaker than a determined adversary's would be. The corrected figure is a **floor** on what is achievable, not a bound on what is possible. Reporting it as "the service is robust" repeats the original sin with the sign flipped. ### What you would check, permanently Make the round-trip an assertion rather than a habit: an example is only counted as a success after it has been serialised through the real ingestion path, re-read, and re-scored. Fail the harness loudly when the wrapper's clean predictions stop matching the live service's, because that is the signal that a CDN transform, a resize filter or an upload limit changed underneath you — and the alternative is a harness that keeps quietly reporting a comfortable number about a pipeline that no longer exists.