Scale Is Not Safety

## The claim this question exists to correct Ask almost any engineer why a model pretrained on a web-scale crawl is safe from data poisoning and you get some version of: *we train on a billion documents; a handful of bad ones average out to noise.* The intuition behind it is not stupid - it is the same intuition that makes label noise tolerable in ordinary supervised learning. It is simply applied to the wrong half of the threat. ## What "dilution" actually assumes Dilution is an argument about **ratios**. It says: the influence of a document on the finished model is roughly its share of the training signal, so if you own 100 documents out of 100 million, you own a millionth of the model and you can move nothing that matters. That argument is sound *for an attacker whose goal is stated as a global property* - overall accuracy, overall completion quality, calibration across the board. Making a model measurably worse everywhere means competing with everything the model learned, so the attacker's requirement is naturally expressed as a percentage of the corpus. Ten times the corpus, ten times the writing, and there is a corpus size past which that attacker cannot afford the attack. This is real, and it is the honest half of the "scale helps" claim. ## The goal where the ratio is the wrong unit Now change the attacker's goal. They do not want a worse model. They want the model to do one specific thing on inputs **they** control: a rare identifier, an unusual configuration key, an obscure package name, a phrase almost nobody writes - and in that narrow context, they want the model's output bent their way. Everywhere else the model should behave exactly as before, because a behaviour change nobody notices is the point. What competes with the planted documents now? Not the billion. A model fitting a rare pattern is effectively fitting it against the other evidence *about that same rare thing*, and "rare" is precisely the statement that there is very little of it. If the corpus contains a dozen documents that mention the identifier at all, a small number of additional ones is not a millionth of anything - it is a large share of the only evidence in existence. That is why this class of requirement behaves as an approximately **absolute sample count** rather than a fraction, and why it is barely diluted by corpus growth: growth adds documents about other things. There is a second-order point that cuts the same way. Growing model capacity alongside the corpus, which is what actually happens between training runs, makes rare patterns *easier* to fit, not harder. So the change that was supposed to buy safety can point mildly the other way. ## The attacker's limit, stated properly This is still a bounded adversary, and saying what bounds them is most of a good answer. They have **no model access at all** - no query, no gradients, no weights, and no knowledge of which snapshot of the crawl was taken. What they have is cheap or free write access to the slices of the public web that anyone can write: pages and repositories they create and control. Their limit is the number of such documents they can place and keep live at negligible cost, and whether that number is enough. For the global-degradation goal it is a ratio against corpus size and it becomes unaffordable as the corpus grows. For the targeted goal it is a count against the local evidence for one rare context and it stays affordable no matter how large the corpus gets. So "is our dataset big enough to be safe?" has two answers pointing in opposite directions, and which one is right depends entirely on which attacker you were asked about. ## What follows for defence If corpus size is not the control, what is? Three things, none of which are size: - **Provenance per document.** Knowing where each document came from is what lets you answer any question after the fact. A crawl with no retained origin cannot be investigated at all. - **A pinned snapshot.** Freezing and storing the exact corpus a run trained on makes it re-fetchable and diffable against the next one. Re-crawling per run means you can never say what changed. - **Behavioural evaluation on what you care about.** You cannot audit a billion documents, so the practical test is against the finished model, on the contexts whose corruption would actually hurt. Sanitization filters and near-duplicate removal help, but they help mostly against the bulk, noisy variant - the one scale already made expensive. ## Saying it in an interview The strong answer is one sentence of concession and one of correction. Concede that dilution is real for an attacker who wants the whole model worse. Then say that an attacker who wants a single behaviour on a rare context has a requirement that does not scale with corpus size, so "we train on billions of pages" is a statement about the first attacker offered as an answer about the second.