How do NTK-aware scaling, YaRN and LongRoPE rescale RoPE frequencies differently?

## Why RoPE needs rescaling at all RoPE gives each dimension pair a wavelength, spread geometrically from a few tokens to far longer than any training sequence. During training at length L, a fast dimension completes many full cycles and the network sees every angle it can take. A slow dimension whose wavelength exceeds L never completes a cycle: it only ever sweeps an arc. Push the model to position 4L and those slow dimensions occupy angles no weight ever encountered. The geometry is still valid; the *input distribution* is not. This is the mechanism behind the familiar observation that a model past its trained length keeps producing fluent text while its accuracy on anything requiring precise reference collapses. Every scheme below is an answer to the same question: how do you map far-out positions back onto angles the model already understands, and which resolution do you sacrifice doing it? ## Position interpolation — the uniform baseline The simplest fix divides every position index by a scale factor s, so a model trained to L can accept sL tokens with all angles landing inside the familiar arc. It is uniform: every dimension is compressed identically. The cost is concentrated where you can least afford it. The fast dimensions were the ones distinguishing "one token back" from "three tokens back"; after dividing by 8, those distinctions are squeezed into an eighth of the angular space they had. Local resolution — the model's sharpest positional signal — degrades to buy long-range range. ## NTK-aware scaling — spend the compression where it is cheap The NTK-aware family reframes the problem: rather than scaling positions, scale the frequency schedule itself by increasing the rotary base. Because frequencies are geometric in the dimension index, raising the base leaves the fastest dimensions nearly unchanged while stretching the slowest ones substantially. High-frequency detail is preserved and the interpolation burden is pushed onto the low-frequency dimensions, which were the out-of-distribution ones to begin with. A dynamic variant computes the scale from the *current* sequence length rather than a fixed maximum, so short prompts are processed with the original schedule and only long ones are rescaled. That avoids paying an extension penalty on requests that never needed it. ## YaRN — make the band split explicit YaRN turns the NTK intuition into an explicit rule, sometimes called NTK-by-parts. For each dimension it computes how its wavelength compares to the training context: a wavelength far shorter than the context means the model saw the full cycle and can safely be left alone, so that dimension is *not* interpolated. A wavelength exceeding the context means the dimension is genuinely out of distribution past L, so it is interpolated fully. In between, a ramp blends the two treatments so there is no discontinuity across the spectrum. YaRN adds a second, separate ingredient: a temperature correction applied to the attention logits, scaling the extension factor logarithmically. The motivation is that spreading attention over a much longer sequence changes the entropy of the softmax distribution, and correcting for that recovers accuracy independently of the frequency treatment. Practically, YaRN is reported to need far less continued training than uniform interpolation to reach the same quality — that efficiency, not just the final score, is much of why it was adopted. ## LongRoPE — search instead of derive LongRoPE observes that no closed-form rule is guaranteed to be optimal per dimension, and replaces derivation with search: an evolutionary algorithm looks for a separate rescale factor for every rotary dimension, scored against an evaluation objective rather than a formula. It also treats the first handful of positions specially, leaving them unscaled, since the earliest tokens play an outsized role in attention. Combined with progressive, staged extension it reached window sizes far beyond what uniform interpolation supported. Later work in this line replaced perplexity-style objectives with retrieval-driven ones, because perplexity turned out to be a weak proxy for whether long-range reference actually works. The tradeoff is plain: search costs compute and produces a vector of numbers with no interpretable rule, but it is not bound by anyone's assumption about which dimensions matter. ## How to reason about the choice The axis running through all four is *how non-uniformly you treat the frequency spectrum*: uniform (interpolation), formula-driven non-uniform (NTK-aware), band-wise non-uniform with an attention correction (YaRN), and freely searched per dimension (LongRoPE). Generality rises, so does the cost of arriving at the coefficients. In mid-2026 these remain the standard vocabulary in open-weight stacks; frontier models increasingly bake long-context handling into a dedicated mid-training stage instead of retrofitting a scaling rule. One caveat to state out loud: none of these are free. Every one trades some positional resolution for range, and a model whose window was stretched by a scaling rule is not equivalent to a model trained natively at that length.