Does scaling data and model size close the clean-accuracy gap that training against a fixed perturbation budget opens?

## The claim being tested A very common senior answer is 'the robustness–accuracy trade-off is a data and compute problem; scale it up and it goes away.' The correcting fact is narrower and more useful: **more data and more capacity narrow the gap and do not close it.** The interviewer is checking whether you understand that hardening changes the *objective*, not just the difficulty of hitting the old one. ## What the harder objective actually is Take a model behind a marketplace ranking surface, where the adversary is a seller who can edit the fields of their own listing within a bounded set of moves the platform permits. Two fits are available: - **Fit the point.** For each training example, be right on the example as it was submitted. - **Fit the neighbourhood.** For each training example, be right on *every* variant of it that the seller could reach inside the permitted edit set. The second contains the first — every constraint of the first is still there, plus a great many more. It is a strictly stronger requirement on the same function class, and strictly stronger requirements do not come free. That is the whole mechanism, and it is worth saying in exactly those terms in an interview. ## Why it is not only a sample-size problem There are two separable reasons the gap exists, and conflating them is what produces the wrong answer. 1. **A statistical reason, which scale does attack.** Learning a function that is stable over regions plausibly needs more examples than learning one that is right at points. This part genuinely improves with more data, including cheaply-obtained unlabelled or weakly-labelled data, and with more capacity. This is why the gap *narrows*. 2. **A structural reason, which scale does not attack.** On distributions that can be written down explicitly, the most accurate predictor and the most robust predictor are *different functions* — the accurate one leans on features that carry real signal but sit inside the adversary's reach, and the robust one cannot use them. No amount of data recovers accuracy you gave up by refusing to use a genuinely predictive feature. This is why the gap *does not close*. Both are true at once. The honest summary is: scale buys you a better position on the frontier; it does not delete the frontier. ## Capacity is not free anywhere There is a second, more practical half to the answer. Even where extra capacity does help, that capacity has a bill of its own: the hardened training run costs substantially more compute than an ordinary one because each step has to consider worst cases rather than the given example, and a larger model costs more at serving time on every request. So 'just scale it' is not a free move even in the regime where it works — you are paying training compute and serving cost to buy back accuracy points, on top of the accuracy points you are still short. ## Direction errors to avoid - **Do not say the gap is an artefact of poor tuning.** It is reproducible across model families and data scales; treating it as a hyperparameter search failure wastes weeks. - **Do not read a narrowed gap as a closed one.** A report showing a hardened checkpoint within a hair of the baseline's clean score, on aggregate, is exactly the case where you ask for the per-slice numbers — the loss concentrates on rare classes, thin-data segments and marginal cases, and an aggregate can hide it. - **Do not generalise across budgets.** Whatever accuracy you bought back was bought at one stated perturbation set. Robustness to one family transfers poorly to another, and a claim quoted without the family and its size is not a claim at all. - **Do not confuse this with an ordinary capacity argument.** Capacity, underfitting and overfitting as learning theory are a separate topic; here the point is specifically that an adversary's reach around each input adds constraints that a non-adversarial fit never faced. ## The answer in one paragraph 'Scaling narrows it. It does not close it, because hardening is a different objective, not a harder instance of the same one: the model must be right across a region rather than at a point, and it has to give up features that are predictive on honest traffic precisely because they are cheap for the adversary to move. Extra data and capacity buy back part of the loss, and they cost training and serving compute to do it. I would plan for a residual gap and price it, not plan to eliminate it.'