For a 1M-token window, do you pretrain natively long or extend a checkpoint post-hoc?

## Why this is a judgment question, not a lookup Interviewers ask it because it forces you to hold three separate budgets in mind at once — training compute, data availability, and serving economics — and because the tempting answer ("just apply a scaling recipe, it worked for 8K to 32K") fails at this magnitude for reasons that are not obvious until you cost it out. ## Budget one: training compute Attention cost grows superlinearly with sequence length, so the per-step cost of training at 1M is enormous relative to training at a few thousand tokens. Meanwhile the overwhelming majority of what a model needs to learn — grammar, world knowledge, reasoning patterns, code idiom — is learnable from short samples. Spending the pretraining budget at maximum length would buy almost nothing extra for most of the tokens and would slash the total token count you could afford. This is why the standard pipeline now has a distinct **mid-training** stage between pretraining and post-training: pretrain the bulk at short-to-moderate lengths, then run a comparatively small stage specifically on long sequences, applying position rescaling as part of it. It captures most of the long-context benefit for a small fraction of the compute. "Natively long" in practice almost always means this staged approach with the architecture designed for length from the start — not full-length training throughout. ## Budget two: data The more binding constraint at 1M is that the data does not exist. Genuinely coherent million-token documents are vanishingly rare — even large repositories and long books fall far short, and the ones that qualify are heavily skewed in domain. So long training samples have to be **manufactured**: pack related documents together so that later parts genuinely depend on earlier ones, concatenate a repository with its history and its issue tracker, synthesize tasks whose answer requires combining facts placed far apart. If you skip this and pad with unrelated text, the model sees long positions but never needs to attend across them, so it learns nothing about long-range dependency and you get a nominal window with no usable capability behind it. That failure mode — high advertised length, weak actual performance — is common and is exactly what modern multi-hop long-context evaluations expose. ## Budget three: architecture and serving This is where the post-hoc option really breaks down. Renumbering positions changes what the model *accepts*; it does nothing to what a long prompt *costs*. A dense-attention checkpoint stretched to 1M still pays full attention cost over a million tokens on every request, plus the per-token state that must be held throughout. For most products the resulting per-request economics are simply not viable, regardless of quality. Making very long contexts affordable is an architectural decision taken at design time: how the attention is structured, whether spans are local with periodic global mixing, whether the per-token state is compressed. Those choices are baked into the weights. You cannot retrofit them onto a shipped checkpoint with a scaling recipe. ## Budget four, usually forgotten: evaluation Whoever ships a 1M window owns proving it works. Single-fact retrieval tests are close to worthless at this scale — they are passed by models that fall apart on anything requiring several linked facts. You need multi-hop and multi-needle evaluation at your target length, and you must expect the honest result: performance at 1M typically sits far below performance at a fraction of that length. Budget the eval work as part of the decision, because a window you cannot certify is a liability. ## How to actually decide A defensible framing: - **Modest extension (a few times the original), existing checkpoint, dense attention.** Post-hoc scaling plus a short long-data stage is the right instrument. Cheap, well-understood, effective. Measure the short-context regression and decide whether to accept it. - **Very long target (tens to a hundred times), you control training.** Design the architecture for the length, pretrain short, run a real long-context mid-training stage on constructed long data. This is what shipping a long-context model actually looks like. - **Very long target, you do not control training.** Do not stretch your way there. Adopt a model built long, and spend the saved effort on the system layer instead — retrieval, compaction, offloading content by reference, sub-agent isolation — which is usually where the win is anyway. ## The strategic point worth making out loud Length is rarely the actual requirement. "We need 1M tokens" almost always decomposes into "we need the model to act on a large body of material", and a system that retrieves and compacts the relevant slice usually beats one that pastes everything in — on cost, on latency, and frequently on accuracy, since a long prompt full of marginally relevant material degrades reasoning. A principal-level answer names the extension tradeoff correctly *and* challenges whether the length target was the right requirement to begin with.