What is the data wall in LLM pretraining, and what options remain when it binds?

## What the wall actually is Every scaling law is a statement about a joint budget: reach a target loss by spending FLOPs on parameters and tokens together. The laws are silent about where the tokens come from, because when they were fitted, data felt effectively unlimited relative to the compute anyone could afford. That relationship has inverted. Compute keeps growing with capital investment and hardware generations. The stock of high-quality, human-written public text does not — it grows at roughly the rate humans write, which is slow, and much of what exists is duplicated, low-quality, machine-generated, or behind licences. Frontier pretraining runs now consume tens of trillions of tokens, which is a serious fraction of the usable public web. The **data wall** is the regime where you can afford more compute than you have distinct, useful tokens to spend it on. The practical symptom is that the two axes stop being interchangeable. Below the wall, "we have more compute" implies "we can train a better model". At the wall, more compute buys you a larger model on the same data — which pushes you toward the undertrained side of the isoFLOP curve, or toward repeating what you already have. ## Lever one: repeat the data The cheapest response is more epochs. Empirical work on data-constrained training found that repeating a corpus for a small number of epochs is nearly as good as an equal quantity of fresh tokens, with the value of each additional pass dropping off sharply after a handful. Push far past that and you get the classic failure: the model memorises rather than generalises, and validation loss stops tracking capability. So repetition buys you a factor, not an order of magnitude. ## Lever two: generate the data This is where the industry actually went. Synthetic generation covers several distinct techniques with different risk profiles: - **Rephrasing and augmentation** — rewriting existing documents into other styles, difficulty levels or formats, which multiplies effective tokens while keeping the underlying facts anchored to real source text. - **Persona- or prompt-diversified generation** — generating large volumes of varied text by systematically varying who is speaking and about what, to avoid the collapse into a single register that naive generation produces. - **Distillation from stronger models** — using a capable model to produce training text, including worked reasoning, for a smaller or newer one. The well-documented failure mode is model collapse: train recursively on your own unfiltered output across generations and the distribution's tails vanish, variance shrinks, and quality degrades. What makes synthetic data work in practice is that it is not recursive and not unfiltered — it is generated by a stronger model or anchored to real documents, then filtered and verified before it enters the corpus. Synthetic data that can be *checked* (code that compiles, maths with a known answer, claims with a source) is far more valuable than synthetic data that can only be judged. ## Lever three: broaden what counts as data Beyond public English prose there is code, many other languages, licensed and proprietary corpora, transcribed audio and video, and images. Each expands the token pool, and some of it is qualitatively different rather than merely additional — code in particular carries structure and verifiability that plain prose does not. Acquiring it is as much a legal and commercial problem as a technical one, which is part of why data access has become a strategic asset rather than a commodity input. ## The deeper consequence The data wall is one of the reasons the field stopped telling a single-curve scaling story. When fresh pretraining tokens are the scarce input, the marginal return on pretraining compute falls, and other regimes — reinforcement learning after pretraining, and spending compute at inference time on a single hard request — start to look like better places to put the next accelerator. Those regimes have their own scarce inputs (verifiable tasks and reward design; latency and per-request budget), but they are not gated on the supply of human prose. ## What a good answer sounds like Name the inversion — compute grows, high-quality text does not — say that the binding constraint has moved from FLOPs to tokens, list repetition, generation and modality expansion with an honest account of what each buys, and note that generation is only safe when the generator is stronger than the student or the output can be verified. Avoid the two lazy poles: "we have already run out" and "synthetic data solves it". Neither is true.