What did the Chinchilla result change about splitting a fixed LLM training budget?

## The problem being solved Pretraining a large language model consumes a fixed, expensive budget of floating-point operations (FLOPs). That budget can be spent in two very different ways: on a bigger model (more parameters, N) or on more training data (more tokens, D). A useful approximation ties the three together: **C ≈ 6 × N × D** where C is training FLOPs. The factor of ~6 comes from roughly 2 FLOPs per parameter for the forward pass and ~4 for the backward pass, per token. The consequence is that N and D trade off directly: at a fixed C, doubling the parameter count halves the number of tokens you can afford. "Scaling laws" are the empirical answer to how that trade should be made. ## Kaplan-era guidance The 2020 scaling-law work fitted smooth power laws relating validation loss to N, D and C, and reported that loss improves predictably as each grows. Its allocation advice was strongly parameter-biased: given more compute, grow the model aggressively, feed it comparatively few tokens, and stop training early rather than run to convergence. The models built on that advice were very large and, by later standards, thin on data. ## The isoFLOP correction The 2022 Chinchilla work asked the allocation question directly rather than inferring it. For each of many fixed compute budgets, it trained a family of models spanning small-and-data-rich to large-and-data-poor, and plotted final loss against parameter count. Each such curve — an **isoFLOP curve** — is U-shaped: too small a model underfits the budget, too large a model never sees enough tokens, and the minimum sits in between. Connecting the minima across budgets gives the compute-optimal frontier. The answer was that N and D should each scale as roughly the square root of compute. Double the FLOP budget and you want about 1.4× the parameters *and* 1.4× the tokens — not 2× the parameters. At the scales studied, the optimum sat near **20 training tokens per parameter**. The headline demonstration: a 70B-parameter model trained on 1.4T tokens outperformed contemporaries several times larger that had been trained on a few hundred billion tokens. The extra parameters had not been wasted so much as starved. ## Why the two results disagreed The discrepancy is now mostly attributed to methodology rather than to a change in the underlying physics. The earlier study used a learning-rate schedule that was not re-tuned to each run's actual token count, which systematically penalises the longer, data-rich runs, and counted parameters in a way that shifted the small-model end of the fit. Re-analyses that correct for these bring the two families of results substantially closer. This is a useful lesson in itself: a scaling law is an empirical fit, and it inherits every training-recipe assumption baked into the runs it was fitted on. ## What the law does and does not tell you It predicts **pretraining loss** — average next-token cross-entropy on held-out text. It does not directly predict downstream task accuracy, instruction-following quality, or reasoning ability, all of which are mediated by post-training. It also says nothing about *serving* cost: it minimises training FLOPs for a target loss, treating inference as free. That single omission is why the compute-optimal point is usually the wrong place to build a production model. The law is also fitted to a particular data distribution and architecture family. Change the data quality, the tokenizer, or the architecture and the coefficients move; the *shape* (smooth power law, U-shaped isoFLOP curves) has proved robust, the constants have not. ## Status of 20:1 today As of mid-2026, ~20 tokens per parameter is a historical reference point, cited to explain the correction, not a target anyone hits. Production models are routinely trained at hundreds to thousands of tokens per parameter — an open-weight 8B model trained on 15T tokens sits near 1,900:1. Two forces pushed practice past the frontier: inference economics, which reward small models even at extra training cost, and the fact that tokens rather than FLOPs are increasingly the binding constraint. Meanwhile the single-curve story has itself been superseded: pretraining is now discussed alongside reinforcement-learning post-training compute and test-time compute as three separate scaling regimes with different returns. ## What an interviewer is checking That you know the allocation question exists, that the answer is empirical and was revised, that ~20:1 was the revised answer at the time, and — most importantly — that you can say why nobody builds at that ratio any more.