Your Counterfit scan target points at a live production inference endpoint instead of a locally loaded model. What does that change about what the scan result is actually measuring, and about your ability to repeat the run?

**What changed under the scan.** A locally loaded checkpoint is a function: same input, same weights, same answer, free to call. A production inference endpoint is a *system* — request validation, whatever resize, normalisation or tokenisation the service applies, an input filter or moderation hop, a served artefact that may be a quantised or distilled copy of the checkpoint your data science team evaluates, a response formatter that may round or trim scores, a cache, and an autoscaler that decides which replica answers. Your Counterfit target's `predict` reaches all of it. Every attack result therefore characterises that whole pipeline at that hour, not the model. That is not automatically a downgrade. If the question is "can a customer with an API key make this service misclassify?", the deployment is exactly the right thing under test, filters and all, and a local scan would answer a different question. The mistake is answering one question and reporting the other. **What breaks repeatability, roughly by how often it bites.** - **Preprocessing mismatch.** The target declares one input space and the service applies another. The attack optimises in a space the model never sees, and *every* attack under-performs uniformly. The reported success rate drops, and a low success rate reads as robustness. This is the single most misleading failure in the list because it looks like good news. - **Score degradation.** The service returns two decimal places, or only the top three classes. A score-based search still runs, just badly, plateauing far below what the same attack achieves on a local copy of the same architecture. No error is raised. - **Version drift mid-scan.** A deploy lands at 02:00 and the back half of an overnight battery hits different weights. Nothing in the result file records it unless your `predict` logged a deployment or model version alongside each response. - **Caching.** Identical repeated queries answer instantly and identically. Convergence looks suspiciously smooth, and any attempt to measure the service's nondeterminism is meaningless. - **Errors read as robustness.** Throttled, timed-out or 5xx calls come back as unusable predictions. Unless the wrapper distinguishes an error from a prediction, the attack simply records no success and the scan silently counts a transport failure as a model that held. **What it costs.** Beyond the per-call price, a live target costs latency you cannot compress — rate limits and backoff stretch a scan that would take minutes locally into hours — and it costs other people's attention. Your queries land in production telemetry and will page whoever watches anomaly dashboards if you did not tell them. If the service logs inputs for future training or evaluation, you have just injected adversarial samples into someone's data pipeline. Both belong in the rules of engagement before the first scan, not in the report afterwards. **Where the number misleads, stated precisely.** A per-attack success rate off a production endpoint means: *with this attack, at these arguments, against this deployment, in this window, on these seeds, this fraction flipped.* Every qualifier is load-bearing, and a live target is what makes the last three move underneath you while you measure. Two specific misreadings to name: quoting the figure as a property of the model (it is a property of a deployment, and a filter in front of the model may be doing all the work), and quoting a *low* figure as evidence of robustness when the plausible causes — wrong input space, degraded scores, error responses counted as failures — all push the number down without telling you. **What I would check, and what goes in the write-up.** Before: send a handful of clean seeds to both the endpoint and a pinned local copy and compare predictions and full score vectors; systematic disagreement on clean inputs means stop, the space is wrong. Inspect one raw response for precision and top-k trimming. During: log every raw response so the run can be replayed offline, record a deployment or model version per response where the service exposes one, and count error responses as their own outcome so affected seeds can be re-run rather than silently scored as failures. After: report the deployment identifier and the scan window, the error count beside the success count, whether scores came back at full precision, and an explicit sentence saying the figure characterises this deployment on this date. If the client wants a claim about the model itself, that scan belongs against a pinned staging replica, and the difference between the two runs is itself a finding — it tells you what the serving stack is contributing.