Two image classifiers are scored with the same library metric — the mean perturbation size over the examples an attack flipped — using the same attack, the same norm and the same settings. Model A's number is larger than model B's. Why does that not settle which model is more robust?

**The statistical shape.** The quantity the helper returns is `E[distance | the attack succeeded on this row, model M]`. It is a conditional expectation, and the conditioning event is model-specific: it is "whichever rows *this* model happened to lose". Identical attack settings fix the *procedure* applied to both models; they do not fix the *sample* each average is taken over. Two conditional means whose conditioning events differ are not on a common scale, and no amount of matching norms, budgets or random seeds repairs that. This is why the answer is not "run it again more carefully" — it is a property of what the helper computes. **The concrete inversion, which is the part worth holding onto.** Suppose the attack flips 8% of model A's rows and 85% of model B's. A hardened model resists nearly everything and loses only on genuinely borderline points — points that by construction already sit near a decision boundary and therefore need *small* perturbations. A weak model loses almost everywhere, including on points that were far from any boundary and required *large* perturbations to move. Averaged, the hardened model can score **lower** on mean perturbation while being unambiguously harder to break. Selection into the average runs against the property you are trying to measure, so the ordering of the two floats carries no ordering of robustness — not a weak signal, no signal. **Second effect: thin denominators.** Eight successes out of a hundred rows produce an average with enormous variance; one outlier moves it visibly. Eight hundred successes produce a stable one. A confidence interval on eight points and one on eight hundred are wildly different evidence, and the helper hands you a bare float with no indication of either. Comparing A's eight-row mean to B's eight-hundred-row mean is comparing an anecdote to a measurement. **What I would actually run.** - Freeze one example set and use it for both models, filtered to rows each model classifies correctly at baseline — and record how that filter differs per model, because it has already changed the population before the attack starts. - Compare **attack success rate at a fixed perturbation budget**. The denominator is every eligible row, identical in construction for both models, and the quantity is monotone in the direction you care about: more successes means less resistant, always. - If a distance comparison is genuinely wanted, restrict it to the *intersection* of the two success sets and compare paired distances there. Small, but genuinely like-for-like. - Carry `n_evaluated`, `n_eligible`, `n_flipped` and the budget beside every number so a reader can see how thin either average is. **What that costs.** You are now paying for two full attack runs over a frozen set rather than reusing whatever each team ran last quarter, and the intersection trick shrinks your usable sample — if A loses only 8% of rows, the paired comparison rests on at most those rows, so obtaining a stable estimate may mean growing the evaluation set several-fold and paying several times the GPU-hours or metered queries. A defensible pinned-budget comparison of two models on a vision workload is realistically hours of accelerator time plus a day of engineer time to build and check the harness; a "just diff the two floats we already have" comparison costs nothing, which is exactly why it is the one people do. **Where it still misleads even after you fix the denominator.** A success rate under a single attack family at a single budget is an *upper bound* on robustness under that procedure only. Two models can trade places under a different attack family, and a model tuned against the family you chose will look better than it is. So the honest output is never "A is more robust than B". It is: under this named attack, at this named budget, on this named frozen example set, A's success rate was x% over n rows and B's was y% over m rows — with the explicit note that a different attack family could reverse it. A ranking claim that outlives its attack, its budget and its example set is the failure mode this whole question exists to catch.