Two merge-gate detectors report 55% and 48% robust accuracy against edited diffs - is the first better?

## The mistake this question exists to catch The reflex answer is arithmetic: 55 is bigger than 48, so model A is better. That reflex is wrong for a reason that has nothing to do with statistics and everything to do with what the number is. **Robust accuracy is the score of a contest between a model and one specific attack under one specific budget.** Two numbers produced by two different contests do not rank the contestants any more than two runners' times rank them when one ran uphill. ## Why the weaker evaluation wins the table An attack searches the space of permitted modifications for one the model reads wrongly. A better search finds at least as many as a worse one, so each reported figure is an **upper bound** on the true robustness of that model at that budget. Upper bounds do not compare: the model whose evaluator tried less hard gets the flattering ceiling. In practice the difference is not marginal. Two parameters do most of the moving: - **Optimisation steps.** A search stopped early has explored a fraction of what it would have. Extending steps until the curve flattens routinely takes a reported figure down by tens of points on the same weights. - **Access class.** An attack that can see weights and gradients searches far more efficiently than one working from the returned decision alone. Handing the attacker less access does not make the model stronger; it makes the measurement weaker, and it silently converts a robustness claim into an obscurity claim. **Restarts** matter for the same reason as steps: a single starting point can stall in a poor region of the search space, and sampling several starting points finds failures one run misses. ## What must match before a comparison means anything | Column | Why a mismatch breaks the ranking | | --- | --- | | Attack and access class | A decision-only attack and a full-access attack are different adversaries; their numbers are on different scales | | Optimisation steps and restarts | Sets how much of the truth the number hides; the largest single lever on the figure | | The permitted edits and their count | The budget *is* the adversary; without it, neither figure describes anyone | | Test set and clean accuracy | A different set, or a detector that simply flags more, changes both numbers for reasons unrelated to robustness | Notice that on a source diff there is no radius to quote. The budget has to be stated as which edit operations an adversary could draw from and how many per input. A table where both rows say only "semantics-preserving edits" has stated no budget twice, and the two rows may be describing adversaries that differ by an order of magnitude in what they were allowed to change. ## What a defensible comparison looks like One attack, chosen by whoever is doing the comparing, run against both models at one stated budget and one access class, at a step count pushed until the figure stops falling, with restarts stated, reported beside clean accuracy on the same test set. Anything short of that is a claim about two evaluations rather than about two models. If you cannot do that - because you are reviewing a write-up and hold neither checkpoint nor endpoint - the correct output is not a ranking. It is a list of the columns that must be reported before the table supports one. As a reviewer that is a stronger position than it sounds: the authors chose which attack to run and how long to run it, and "we cannot tell which model is better from this table" is a finding, not an admission. ## The trap on the other side Do not over-correct into treating clean accuracy as the tiebreaker either. Defences that buy robustness generally give up some clean accuracy, so the model with the lower clean figure may be the one that actually did something. The two axes are read together, and neither one alone orders a pair of systems.