How do grouped KV heads, latent compression or recurrent layers change cache bytes per token?

## The formula and the one term that varies All of this is the same arithmetic with one term swapped. For a length-proportional attention layer, cache bytes per token per layer = 2 x kv_heads x head_dim x bytes_per_value. Multiply by layer count for the stack. Model families differ almost entirely in what they put in place of that middle term, and the differences are large — a factor of ten or more between designs at the same parameter count. ## Full multi-head attention: the baseline When every query head owns its own key and value head, kv_heads equals the number of attention heads. On a 32-layer model with 32 heads of head_dim 128 in bf16, that is 2 x 32 x 128 x 2 = 16 KB per layer, 512 KB per token, and roughly 51 GB for a 100,000-token session. That is why long-context serving was impractical in the original dense-MHA era: one session could exceed a whole accelerator. ## Grouped and multi-query: divide the KV heads Grouped-query attention keeps all query heads but has each group of them share a single key/value head. The cache term becomes the number of *groups*. Eight KV heads behind 32 query heads is a clean 4x reduction: 4 KB per layer, 128 KB per token, about 13 GB per 100K session. Multi-query attention is the extreme case of one shared KV head, dividing by the full head count. The compute of attention is essentially unchanged — the shared key/value is broadcast across its group — so this is a pure memory win, paid for at training time in modelling capacity. ## Latent compression: one vector instead of a pair Multi-head latent attention takes a different route: instead of storing per-head keys and values, it caches a single low-rank latent vector per token per layer and reconstructs the per-head keys and values by up-projection inside the attention computation. Because positional encoding cannot be folded cleanly into the compressed part, a small decoupled positionally-encoded key is stored alongside it. With a latent dimension of 512 and a decoupled key of 64, that is 576 values per token per layer — about 1.1 KB in bf16, or roughly 36 KB per token across 32 layers. Compared with the grouped-query baseline above, that is another 3-4x reduction, and against full multi-head roughly 14x. The tradeoff is arithmetic: the reconstruction is extra work at attention time, which lands on the compute side where decode has headroom. ## State that does not grow: recurrent and windowed layers A third family removes the length dependence entirely. Linear-attention and gated-recurrent layers summarize all history into a fixed-size state that is updated per token; the state is the same size at token 10 and token 1,000,000. Sliding-window attention layers keep a genuine key/value cache but only for the last W positions, so their footprint is capped rather than constant-per-token. Neither is normally used alone — a stack of only recurrent layers struggles with precise long-range recall — so mid-2026 production models interleave them, commonly with a minority of full-attention layers among a majority of linear ones. In such a hybrid, only the full-attention layers grow with context: if 8 of 32 layers are full attention at the grouped-query shape above, per-token cost drops to 32 KB plus a constant, roughly a quarter of the all-attention stack. ## Precision multiplies everything The bytes_per_value term applies uniformly. Whatever attention design you pick, holding the cache in an 8-bit format instead of bf16 halves every number above, and 4-bit quarters them. Precision and architecture multiply rather than substitute, so a latent-attention model with an 8-bit cache sits nearly thirty times below a dense multi-head bf16 baseline. ## Reading a model card like a capacity planner When you evaluate a model for a long-context product, the fields that matter for memory are layer count, number of key/value heads (not attention heads), head dimension, whether the stack uses a compressed latent, and whether any layers are windowed or recurrent. Two models with identical parameter counts and identical benchmark scores can differ by an order of magnitude in how many concurrent sessions they let you serve, and that difference usually shows up on the invoice before it shows up in quality. Do the multiplication before the evaluation, not after the incident. ## What this does not decide Cache footprint is one input among several. Sharing key/value heads, compressing to a latent and interleaving recurrent layers all have quality and training implications that belong to the architecture discussion. The arithmetic here tells you what each choice costs at serving time; it does not by itself tell you which model is better.