When would you interleave linear-recurrent layers with full attention in a long-context model?

## Three different escapes from the same cost By the time a model targets very long inputs, full attention's cost is the dominant constraint, and there are three structurally different responses. Head sharing and latent compression shrink the *state* attention retains, but every query still consults every earlier token. Learned sparsity shrinks *what each query consults* while keeping the retained state exact. Linear and recurrent layers do something more radical: they abandon the pairwise structure altogether and maintain a fixed-size running state that is updated as tokens arrive. That third option is qualitatively different, because it changes what the layer is *capable* of, not just what it costs. ## What a fixed-size state buys and what it costs A recurrent layer's state does not grow with sequence length. Processing the ten-thousandth token costs the same as processing the tenth, and the memory held between steps is constant. Over very long sequences that is a categorical win, not a constant-factor one. The cost is **exact recall**. Attention can, in principle, reach back and read any single earlier token verbatim, because that token's representation is still individually present. A fixed-size state cannot — everything has been mixed into a summary of bounded capacity. Modern recurrent designs such as Mamba-2 and gated delta-rule layers make the state update input-dependent, so the layer learns what to keep and what to overwrite, and that closes much of the gap on tasks where the relevant information is compressible. It does not close the gap on tasks that require pulling out one arbitrary detail from far back, and long-context retrieval evaluations show that clearly. A related misconception worth pre-empting: these layers are not slow to train. Their recurrence has a form that can be reassociated into chunked parallel computation, so training parallelism is preserved. The tradeoff is about representational capacity, not about trainability. ## Why hybrids rather than pure stacks Because the two mechanisms fail in complementary ways, mixing them dominates either extreme in practice. A stack that is entirely recurrent is cheap but loses precise lookup. A stack that is entirely attention is precise but pays the full long-context cost at every layer. Interleaving — commonly around three recurrent layers for each full-attention layer, with the exact ratio varying by model family — lets the recurrent majority carry the sequence cheaply while the periodic attention layers restore exact access. A small number of attention layers turns out to be enough to recover most retrieval ability, which is why the ratio is skewed rather than balanced. ## Making the decision Start from the retrieval profile of the workload, because that is what discriminates. If the task is *aggregative* over a long sequence — detecting a diffuse signal across a hundred thousand bases of genomic sequence, summarising a long continuous stream, modelling long-range statistical structure — the fixed-state summary is a good match, and a hybrid pays. If the task is *retrieval-shaped* — find the clause, cite the line, recall the exact identifier — you want exact attention over the whole input, and the right lever is one that preserves exactness: head sharing, latent compression, or learned sparsity, which reduce cost without giving up individual token access. Then weigh the commitment. A hybrid stack cannot be bolted onto a trained model; the mixture must be present during pretraining. That means the decision is made before you have evidence at full scale, and small-scale ablations do not always predict where the recall cliff lands. It also means a serving-stack tax: recurrent layers need their own kernels and their own state handling, and every tool in the pipeline must support the mixed architecture. Finally, be honest that this is contested ground. As of mid-2026 several model families ship hybrid stacks and report strong results, while others reach the same context lengths with attention plus learned sparsity. There is no settled consensus about which wins, and a principal-level answer should say so rather than assert one. ## What evidence to demand Before committing, insist on long-context evaluations that stress *multi-hop and multi-needle* retrieval, not a single-needle probe and not average loss. Aggregate metrics hide exactly the capability a fixed-size state sacrifices, and a model can look fine on perplexity while failing the retrieval behaviour your product depends on. Also insist on measurement at the context length you actually intend to serve, since the degradation is non-linear and often appears well below the advertised limit. ## Answering it well Lead with the retrieval-profile question, since that is the real discriminator. Explain the constant-state tradeoff honestly in both directions, justify the interleaving as complementary failure modes rather than as a compromise, and name the commitment: pretraining-time, non-retrofittable, with a failure mode that average metrics do not surface. Saying plainly that the field has not settled this is a strength, not a hedge.