Why does every Llama 3 size use grouped-query attention rather than multi-head attention?

## The problem GQA solves Autoregressive decoding generates one token at a time. For each new token, attention needs the key and value vectors of every token that came before it. Recomputing them would be quadratic waste, so serving stacks cache them — the **KV cache**. The cache is per-request, grows linearly with sequence length, and lives in accelerator memory alongside the weights. Size it yourself. Bytes per token equals `2 (K and V) x layers x kv_heads x head_dim x bytes_per_element`. For Llama 3 8B — 32 layers, 8 KV heads, head dimension 128, bf16 at 2 bytes — that is `2 x 32 x 8 x 128 x 2 = 131,072` bytes, i.e. 128 KiB per token. A single 128K-token conversation therefore parks about 16 GiB of KV cache. Now redo it with 32 KV heads, as plain multi-head attention would require: 512 KiB per token, 64 GiB for that same conversation. On a single 80 GB accelerator that alone decides whether the deployment is possible. ## What grouped-query attention actually is Three points on a spectrum: - **Multi-head attention (MHA).** Every query head has its own key head and value head. Maximum expressiveness, maximum cache. - **Multi-query attention (MQA).** All query heads share exactly one key/value head. Minimum cache, and measurably worse quality, with training instability reported at scale. - **Grouped-query attention (GQA).** The middle: query heads are partitioned into `G` groups, and each group shares one key/value head. `G = num_heads` reduces to MHA; `G = 1` reduces to MQA. GQA keeps the *query* projections untouched — the model still has 32 or 64 distinct query heads attending in 32 or 64 different ways. Only the key and value projections are narrowed. Empirically that costs very little quality while recovering nearly all of MQA's memory savings, which is why it has become the default across essentially every open-weight family, not just Llama. ## The Llama-specific facts In a Hugging Face `config.json` the two fields to read are `num_attention_heads` and `num_key_value_heads`; their ratio is the group size. For Llama 3 and 3.1: - **8B**: 32 attention heads, 8 KV heads, 32 layers, head dim 128 → group size 4. - **70B**: 64 attention heads, 8 KV heads, 80 layers → group size 8. Note the KV head count stays at 8 as the model scales. That is deliberate: the larger model gets more query heads and more layers, but the per-layer KV footprint does not grow proportionally, so cache pressure scales more gently than parameter count would suggest. Llama 2 is the useful contrast. Its 7B and 13B used ordinary multi-head attention; only the 70B adopted GQA. Making GQA universal in Llama 3 was a serving decision as much as a modelling one — it is a precondition for the 128K context that arrived in 3.1, because without it the cache for a full window on the small model would dwarf the weights. ## Why the saving shows up as latency, not just memory Token-by-token decoding does very little arithmetic per byte moved: for each generated token you stream the weights and the entire KV cache through the accelerator's memory system. That makes decode **memory-bandwidth-bound**, not compute-bound. Shrinking the KV cache by 4x removes 4x of the bytes that must be read per step from the cache side, which shows up directly as higher tokens-per-second — especially at long contexts, where the cache dominates the weights in bytes moved. It also raises the batch ceiling. Concurrency in a modern server is limited by how many requests' caches fit in memory at once. Quarter-size caches mean roughly four times the concurrent sequences at the same memory budget, which is throughput and cost per token, not just a headroom nicety. ## Where the tradeoff bites GQA is not free. Sharing key/value heads reduces the diversity of what each group can attend over, and ablations show a small quality cost relative to full MHA — small enough that every major family accepted it. Models are trained with their group ratio from the start; you cannot convert an MHA checkpoint to GQA at serving time without an uptraining procedure, so the ratio is a fixed property of the weights you downloaded, readable from the config and nothing you tune. ## Answering well Lead with the mechanism (query heads grouped over shared KV heads), give the Llama numbers (8 KV heads at every size, 4:1 on 8B, 8:1 on 70B), then explain *why anyone cares* — the KV cache is the thing that scales with context and concurrency, decoding is bandwidth-bound, and the saving converts into longer windows and larger batches. Candidates who describe GQA purely as a parameter-count optimisation have missed the point; the weights barely shrink, the cache does.