Why would a long-context model leave some head dimensions unrotated by RoPE?

## Partial rotation is older than the current debate It is easy to assume RoPE applies to a whole head. It never had to. Some of the first models to adopt rotary encoding rotated only about a quarter of each head's dimensions and left the remainder alone, and it worked. What has changed by mid-2026 is that the split stopped being an implementation detail and became an explicit architectural lever, discussed alongside NoPE — running with no positional encoding whatsoever and relying on the causal mask's implicit signal. ## The mechanism: rotation entangles matching with distance A rotated dimension pair contributes a term to the query-key dot product that varies with the angle between the two tokens' rotations — that is, with their offset. Sum many such planes across the fast and slow frequency bands and, for generic vectors, the aggregate contribution tends to oscillate and attenuate as the gap grows. For most of language this is a desirable prior: nearby tokens usually matter more, and the model gets that for free. But it is a prior imposed on *every* head in *every* layer. Some heads exist to do long-range retrieval — locate the single stack frame, clause, or record that matters, wherever it is. For those, distance-dependent attenuation is noise fighting the objective, and the head must spend capacity learning to overcome its own positional machinery. An unrotated subspace removes that fight. In those dimensions the dot product is a pure content match: a query and a key that agree score the same whether they are 12 tokens apart or 120,000. The model gets both behaviours in one head — locality where rotation applies, distance-blind matching where it does not — instead of one compromise. ## The variants you will hear named **Dimension splits.** Rotate a fraction of each head's dimensions; leave the rest position-agnostic. The fraction is a hyperparameter with no consensus value. **Layer splits.** Keep some layers fully rotary and make others position-free, interleaved through the stack, so early layers can establish local structure while designated layers do distance-blind retrieval. **Full NoPE.** No positional encoding at all, relying on the causal mask giving each position a differently sized visible prefix. Research showed decoder-only models can learn order this way and can generalize to longer inputs better than encoded ones on some tasks. It is fragile where exact local ordering matters, so pure NoPE at frontier scale remains contested. Separately, some compressed-KV attention designs are *forced* into a split for cache reasons: when keys are reconstructed from a compressed latent, rotation cannot be applied to the compressed form, so a small dedicated set of dimensions carries the position signal and the rest carries content. That is a mechanical constraint, not the retrieval argument above — worth distinguishing, because they arrive at superficially similar architectures for entirely different reasons. ## The costs, stated honestly Withdrawing rotation withdraws the model's sharpest signal for exact local ordering. The causal mask supplies order only weakly and indirectly, and tasks that hinge on precise adjacency — parsing nested structure, exact sequence reproduction, code with strict token order — are where an over-aggressive split shows up. It also complicates reasoning about the model: two subspaces with different positional semantics is harder to analyse and to debug than one uniform rule. And there is no formula. The split ratio, the layer pattern, and whether to do it at all are empirical calls made per architecture, and mid-2026 stacks disagree with each other. Anyone presenting a specific number as settled best practice is overstating the state of the field. ## How you would actually decide Treat it as a hypothesis with two competing evaluations, not one. Measure a long-range retrieval task where the target sits far from the query, and separately measure a task that hinges on exact local order. A good split improves the first without moving the second; a bad one buys retrieval by quietly losing precision that only shows up on structured output. Averaging both into a single score hides exactly the tradeoff you are trying to observe. ## What a principal-level answer sounds like Name the mechanism (rotation couples similarity to distance), name the benefit (a distance-blind subspace for retrieval heads), name the cost (weaker exact local ordering), name the variants (dimension split, layer split, full NoPE), and say plainly that the field has not converged. Confidence about the right ratio is the tell that someone is reciting rather than reasoning.