An adversarial library's metric helper — the one that averages perturbation size over the examples an attack flipped — returns 0.0 after a run in which the attack flipped nothing. Why is that zero not evidence the model is trivially fragile?

**The mechanism.** The helper computes `sum(measured_distances) / count(successes)` over the rows an attack flipped. When the attack flips nothing, the numerator is an empty sum and the denominator is zero. Implementations guard that division rather than raise: the Adversarial Robustness Toolbox's `art.metrics.empirical_robustness`, for example, hands back `0.0` when the success mask is empty. That `0.0` is a placeholder emitted by a guard clause. It is not a measurement, and no quantity about the model was computed to produce it. **Why this particular failure bites so hard.** The metric's polarity is "a bigger number means the attack had to work harder", so a zero renders at the *fragile* end of the scale. Two opposite situations therefore print the same value: "the attack broke everything with an imperceptible nudge" and "the attack broke nothing at all". A dashboard that plots this float over time will draw your most resistant run as your worst one, and a reader who has never opened the helper's source has no way to tell which of the two they are looking at. Nothing inside the number separates them — only the success count does, and the helper does not return it. **Three causes, in the order an operator should rule them out.** 1. **A genuinely unsuccessful attack.** The perturbation budget, step count or number of restarts you granted was too small to cross a boundary anywhere in the batch. That is a statement about *your run*, not about the model. The standard response is to escalate strength until you either obtain successes or reach a strength you can defend in writing as adequate for the threat model you were hired to test. 2. **A broken harness.** The wrapper feeds inputs in a different scaling, channel order or normalisation than the model was trained on; gradients come back identically zero because a preprocessing step is not differentiable or `requires_grad` was never set; the attack optimises against label indices the model does not emit. A *perfectly* clean zero-success run is far more often a wiring fault than an unbreakable classifier, and experienced people treat it that way by reflex. 3. **A defence that hides the signal rather than removing it.** A model whose preprocessing masks or obscures gradients can stall a gradient-based search while remaining perfectly breakable by a decision-based one. Zero successes under one attack family is silent about every other family. **What the run cost you.** By the time the `0.0` appears you have already paid the entire bill — every forward and backward pass, every metered query, every GPU-hour of the sweep — for a value that carries no information. Escalating strength is not free either: doubling the step count roughly doubles the compute, and multi-restart searches multiply it again, so a "just run it harder" reflex can turn a twenty-minute job into an overnight one. That is why the diagnostic order above matters commercially, not just intellectually. The cheapest checks come first: a single forward pass to confirm the wrapper's predictions match what the served model returns, one gradient call to confirm the tensor is not all zeros, and a control run of the same attack against a deliberately undefended reference model. Those cost minutes. If the same attack cannot break the undefended control either, the fault is in your harness and no amount of extra budget will fix it. **Where the number misleads and what I therefore report.** The seductive misreading is the optimistic one: "we ran the toolkit, nothing broke, the model resisted". That sentence upgrades an absence of evidence into a robustness claim, and it is exactly the sentence a zero-success run tempts you to write. The pessimistic misreading is the mirror image: a stakeholder sees `0.0` on a scale where small means fragile and concludes the model collapses under any nudge. Both readings come from quoting a conditional mean without its condition. So I never publish the bare float. I report the number of rows evaluated, the number flipped (zero), the attack family with its budget and search settings, the outcome of the harness sanity checks, and an explicit sentence that the metric is undefined for this run. If a stakeholder needs one number, robust accuracy at a stated perturbation budget has a real denominator — every evaluated row — and does not degenerate when the attack fails.