What does Multi-head Latent Attention compress, and how is that different from quantizing the cache?
answer
- Shape change, not precision change
- One small vector per token, expanded per head
- Learned during training, not applied after
- Linear up-projections can be folded away
- Orthogonal to number-format tricks
basics
~20 sMulti-head Latent Attention projects each token's keys and values down into one small learned latent vector and retains only that. Per-head keys and values are reconstructed on the fly by up-projection. The compression lives in the trained weights, not in the number format.
solid answer
~50 sMLA replaces per-head key and value storage with a **low-rank latent**: a single down-projection maps each token's hidden state into a compact vector, and separate up-projections expand that vector back into every head's keys and values when attention runs. Only the latent is retained across decoding steps, so the state per token is far smaller than even aggressive head sharing gives you. Because the up-projections are linear, they can largely be folded into the query and output projections, so the full keys and values need not be materialised at all. The distinction from quantization matters: quantizing shrinks each stored number's precision while keeping the tensor's shape, and it is applied to an already-trained model. MLA changes the *shape* of what is stored, and the model is trained in that compressed space, so it learns to work with the reduced rank instead of tolerating rounding error after the fact. The two are orthogonal and can be combined.
code
python · 10 linesimport numpy as np
d_model, n_heads, d_head, d_latent, seq = 512, 8, 64, 64, 10
x = np.random.randn(seq, d_model)
W_down = np.random.randn(d_model, d_latent)
W_up_k = np.random.randn(d_latent, n_heads * d_head)
latent = x @ W_down
keys = (latent @ W_up_k).reshape(seq, n_heads, d_head)
print(latent.shape, keys.shape)go deeper
Know the shape of the idea: instead of storing full keys and values for every head, the model stores one small learned vector per token and expands it when needed. That is a change to the architecture, not a storage format applied afterwards.
Be ready to describe the down-projection and up-projection pair, say that only the latent persists between steps, and explain why this is a low-rank factorisation rather than rounding. Naming that the compression is trained in is the key discriminator.
Expect to explain the absorption of the up-projections into the query and output projections, why that keeps decoding cheap, and how the tradeoff differs from a precision reduction you can apply post hoc. Being explicit that the two compose is a strong signal.
Own the adoption decision: MLA is a pretraining-time commitment with a long tail of serving-stack and tooling support, validated only after real compute is spent. Weigh that against a conservative head-sharing baseline plus format-level compression, and be able to say what evidence would move you.
## The problem MLA attacks Grouped-query attention reduces retained key/value state by a constant factor — divide the number of key/value heads by eight and you divide the state by eight. But each surviving key/value head is still a full-width vector per token, and the total still grows linearly with sequence length. For models targeting very long inputs, a constant-factor cut is not enough. Multi-head Latent Attention, introduced in the DeepSeek model line and since adopted by several other families, attacks the width of the stored object rather than the count. ## The mechanism For each token, instead of computing and retaining per-head keys and values directly from the hidden state, MLA first applies a **down-projection** into a latent vector whose dimension is much smaller than the concatenated per-head key and value dimensions. That latent vector is the only thing retained. When attention is computed, **up-projection** matrices expand the latent back into per-head keys and per-head values. This is a low-rank factorisation of the key and value projections, constrained so that the bottleneck sits exactly where storage happens. The rank of the bottleneck is a design parameter, chosen to trade quality against retained state. ## Why it is not just extra work at decode time The obvious objection is that reconstructing keys and values on every step trades memory for compute. In practice the up-projections are linear and can be **absorbed** into neighbouring matrices: the key up-projection can be folded into the query projection and the value up-projection into the output projection, algebraically, before serving. After that folding the model never materialises full per-head keys and values at all — it attends in the latent space directly. That is what makes the design pay rather than merely relocate the cost. One detail complicates the folding: rotary position information does not commute with the absorbed up-projection, so MLA carries a small separate positional component alongside the latent rather than baking position into it. ## Compression as architecture versus compression as post-processing This is the comparison interviewers actually want. Quantizing a cached tensor takes a trained model, keeps every stored value in place, and represents each one in fewer bits. The tensor's shape is unchanged; the information loss is rounding noise the model was never trained to expect, which is why calibration and per-block scaling matter so much and why quality degrades in a hard-to-predict way at aggressive widths. MLA changes the shape. There is no full tensor being approximated at serve time — the compressed representation *is* the model's representation, present during pretraining, and gradients flow through the bottleneck. The model learns which directions of the key/value space are worth keeping. That makes the loss principled rather than incidental: the network allocates its limited rank where it helps. The practical consequences differ accordingly. Quantization can be applied to any released checkpoint after the fact; MLA cannot be retrofitted without retraining, because the bottleneck must be present while the weights are learned. Conversely quantization's savings are bounded by the number format, while a latent's savings are bounded only by how small a rank the model tolerates. The two compose: a latent representation can itself be stored in a reduced-precision format. ## Costs and risks MLA is not free. It adds architectural complexity — extra projection matrices, the absorption trick, the separate positional path — and that complexity has to be right in every serving stack, kernel and conversion tool that touches the model. A team adopting it inherits a support burden that GQA does not carry. It also cannot be validated cheaply: because the bottleneck must exist during training, you find out whether your chosen rank was too aggressive only after committing significant pretraining compute. And the reconstruction is lossy by construction; at small ranks the model's ability to distinguish fine-grained retrieval targets degrades. ## How to answer Lead with the shape change — one small learned vector per token instead of per-head keys and values — then the absorption point that keeps decode cheap, then the contrast with quantization framed as *trained-in versus applied-after* and *shape versus precision*. Close by saying they are orthogonal and stackable. Being honest that MLA is a pretraining-time commitment, not a deployment knob, is the answer's most senior note.
- If the up-projections can be absorbed into neighbouring matrices, why is a separate positional path still needed?Because rotary position information is applied as a rotation that does not commute with the absorbed up-projection — folding the matrices together would destroy the position-dependence. MLA therefore carries a small dedicated positional component per token alongside the compressed latent, so that positional structure survives while the bulk of the key/value content stays in the folded latent path.
- Can you retrofit MLA onto an already-trained model the way GQA can be uptrained from MHA?Not cleanly. The bottleneck has to exist while the weights are learned so the model allocates its limited rank deliberately; imposing a low-rank factorisation after the fact approximates a representation that was never trained to be low-rank. Head-sharing conversion works because mean-pooling similar heads is a mild edit; forcing a much smaller rank is not.
- How would you choose the latent rank for a new model?Treat it as a quality-versus-state sweep at small scale before committing pretraining compute: train several ranks, measure loss and long-context retrieval accuracy, and look for the knee where retrieval quality starts falling faster than the state savings justify. The risk is that small-scale knees do not always transfer to full scale, so leave headroom rather than sitting exactly on the edge.
Quantizing a cache is like saving a photo library as lower-quality JPEGs: same number of images, each blurrier. A latent bottleneck is like training a sketch artist to capture each scene in a handful of strokes: fewer marks on the page, but chosen during training to be the ones that matter.
saying these in an interview costs you the question
- Describes MLA as quantizing the stored keys and values
- Says the latent is computed once and shared across all tokens
- Claims MLA can be applied to any released checkpoint
- Assumes reconstructing keys per step makes decoding slower
- Treats low-rank compression and reduced precision as the same technique