Why deliberately train a 3B LLM far past its compute-optimal token count?

## Two different optimisation problems The compute-optimal frontier answers: *given C training FLOPs, what model reaches the lowest loss?* A deployed system asks something else: *given a quality bar and a traffic forecast, what total cost of ownership do I pay?* Those are different objectives and they have different optima. Write the lifetime cost roughly as: **total ≈ training_flops + requests × tokens_per_request × inference_flops_per_token** Training FLOPs are approximately 6 × N × D, paid once. Inference is approximately 2 × N FLOPs per generated token (forward pass only), paid on every token of every request, forever. The moment `requests × tokens_per_request` grows large, the second term dominates — and note that it depends on N but *not* on D. Training tokens are free at serving time. Parameters are not. ## The concrete shape of the decision Consider a feature that will serve on the order of 400M requests a month. Two candidates reach a similar quality bar: a larger model sitting on the compute-optimal frontier, and a 3B model trained on far more tokens than its budget "deserves". The compute-optimal one is cheaper to train and more expensive on every single one of those 400M requests. Over a couple of years of traffic the extra training spend on the 3B model is recovered many times over. Parameter count drives serving cost through several channels at once, which is why it dominates: - **Weight memory.** Fewer parameters means the weights fit in less accelerator memory, leaving room for larger batches and longer KV caches — or fitting on cheaper hardware entirely. - **Arithmetic per token.** Decode cost scales with active parameters, so latency per token falls. - **Batch density.** More concurrent requests per accelerator directly divides the fixed cost of the machine across more users. - **Deployment surface.** Small enough, and the model runs on-device or at the edge, where the marginal serving cost approaches zero. ## Why over-training works at all The loss-versus-tokens curve at fixed parameter count does not have a cliff. Past the compute-optimal ratio it keeps descending, just more slowly — you are on the flat part of a power law. That means over-training is a smooth purchase: you can decide how much extra training compute you are willing to convert into serving savings, rather than facing a hard wall. Practice has run a very long way down that curve; open-weight models in the 7-8B range trained on 15T tokens sit near 1,900 tokens per parameter, roughly a hundred times the classic reference ratio. The limits are real, though. Returns per extra token shrink, so at some point the marginal training FLOP would have been better spent on a slightly larger model, on post-training, or on data quality. And you cannot over-train past the data you have — which is exactly where the data-availability constraint starts to bind. ## Related levers that serve the same goal Over-training is one way to get a small, strong model, and it is usually combined with others: distilling from a larger teacher, quantising weights to 4-bit formats for serving, and architectures where only a fraction of parameters are active per token so that serving cost tracks *active* rather than total parameters. When someone says "inference-optimal sizing", they usually mean the whole bundle, with over-training as the training-side component. ## How to reason about it in an interview The strong answer is not "small models are cheaper". It is the shape of the trade: training cost is capital expenditure paid once and scales with N × D; inference cost is operating expenditure paid per request and scales with N alone. Therefore the right question is never "what is the compute-optimal model" but "at my traffic volume, where does total cost bottom out" — and at high volume that answer is always smaller and longer-trained than the frontier says. The cases where you should *not* over-train are equally worth naming: low-traffic internal tools, research models whose only job is to establish a capability, offline batch jobs where latency is irrelevant and you can amortise a big model over a scheduled window, and anything where the quality ceiling of the small model simply does not clear the product bar. Sizing is a function of the traffic curve, not a universal preference.