How does Llama 3.1 reach a 128K context when Llama 3 was trained at 8K?

## What RoPE does, briefly Llama has no learned absolute position table. Instead it uses **rotary position embeddings**: before attention, the query and key vectors for a token at position `m` are rotated by an angle `m x theta_i`, where each pair of embedding dimensions `i` gets its own frequency `theta_i`. The frequencies form a geometric series from fast to slow, controlled by a base value (`rope_theta`, 500000.0 in Llama 3 versus 10000.0 in Llama 2). Because attention compares a query at position `m` with a key at position `n`, and the rotations combine, the dot product ends up depending on `m - n` — relative position — which is exactly what you want. ## Why the window does not just extend for free High-frequency dimensions complete many rotations within a few thousand tokens, so they generalise: the model has seen every phase. Low-frequency dimensions rotate slowly, and within an 8K training window they never complete even one cycle. Push a token to position 100,000 and those dimensions sit at angles the model has never encountered. The result is not graceful degradation; attention scores go haywire and output quality falls off a cliff. ## The three families of fix 1. **Position interpolation** — divide all positions by a factor so the new maximum maps back onto the trained range. Simple, but it squashes the high-frequency dimensions too, blurring the model's sense of nearby token order. 2. **NTK-aware / base scaling** — raise `rope_theta` instead, stretching wavelengths non-uniformly. 3. **Frequency-dependent scaling (what Llama 3.1 does)** — treat the dimensions differently by wavelength. Dimensions whose wavelength is short relative to the original context are untouched, so local ordering stays crisp. Dimensions whose wavelength exceeds the original context are divided by the scaling factor, so they stay inside seen phases. A smooth ramp interpolates in between rather than switching abruptly. In the Llama 3.1 `config.json` the block reads roughly: `rope_type` `"llama3"`, `factor` 8.0, `low_freq_factor` 1.0, `high_freq_factor` 4.0, `original_max_position_embeddings` 8192, alongside `max_position_embeddings` 131072. The factor of 8 is the 8K-to-64K stretch; the boundary parameters decide which dimensions count as low- versus high-frequency by comparing their wavelength against the original 8,192-token window. ## Scaling alone is not the whole answer This is the part candidates most often miss. Rescaling the frequencies makes long positions *representable*; it does not make the model *good* at using them. Meta paired the change with continued pretraining on long-context data, increasing the sequence length in stages up to 128K, so the model actually learns to retrieve and reason across the extended range. A model with rope scaling bolted on at load time and no long-context training will pass a trivial retrieval probe and fail at anything requiring synthesis over the window. ## What this means operationally - **The config is the contract.** If your serving stack is told the window is 8,192 — an old config, a truncated maximum length, a wrapper that clamps — you get the short window regardless of the checkpoint. Conversely, forcing a larger maximum than the config's scaling supports produces silently degraded output rather than an error. - **Cost is not flat across the window.** Prefill attention is quadratic in prompt length, so a 100K-token prompt is not twelve times a 8K one — it is far worse. The KV cache grows linearly, and on Llama 3.1 8B a full 128K context is roughly 16 GiB of cache per request in bf16, which usually collapses your batch size to one or two. - **Quality degrades before the limit does.** Advertised windows are the ceiling, not the recommended operating point. Measure your own task at 8K, 32K and 128K rather than assuming the last token is as usable as the first. - **Do not confuse rope scaling with other context-extension schemes.** YaRN, for instance, is a different published method used by other model families; naming it as Llama 3.1's mechanism is a factual slip an interviewer will catch. ## Answering well Start from why extrapolation fails — unseen rotation phases in the slow dimensions — then describe the frequency-selective fix and name the config fields, then add the two things that separate a real practitioner from a paper-reader: that continued long-context pretraining was required alongside it, and that the cost curve, not the config, is what actually limits how much of the window you use in production.