Why can an LLM cache keys and values across decoding steps but not queries?
answer
- what stays the same as you append
- causal mask freezes the past
- queries are consumed once
- append-only K and V per layer
- memory grows linearly with tokens
basics
~20 sUnder a causal mask, a past token's key and value vectors never change as new tokens arrive, so every later step reuses them. That token's query was consumed once, to produce its own output, and is never read again.
solid answer
~50 sDecoder attention is causally masked: position i can attend only to positions up to i. So the hidden state at position i — and therefore its projected key and value in every layer — depends only on tokens 0..i and is frozen the moment it is computed. Appending a new token cannot change it, which makes K and V append-only and perfectly cacheable. The query at position i is different: it is multiplied against the keys once, at the step that produced position i's output, and no future step ever reads it again, so storing it would be dead weight. A decode step therefore computes **one** new query row plus one new key/value pair, appends the pair, and attends over the whole cached set. The cost is memory: the cache grows linearly with tokens, layers and KV heads.
code
python · 11 linesimport torch
def decode_step(x_new, cache_k, cache_v, w_q, w_k, w_v):
q = x_new @ w_q # only the newest token needs a query
k = x_new @ w_k
v = x_new @ w_v
cache_k = torch.cat([cache_k, k], dim=0) # append, never recompute
cache_v = torch.cat([cache_v, v], dim=0)
scores = (q @ cache_k.T) / q.shape[-1] ** 0.5
attn = torch.softmax(scores, dim=-1) # no mask: every key is in the past
return attn @ cache_v, cache_k, cache_vgo deeper
Be able to say that generation reuses the key and value vectors of earlier tokens instead of re-running the whole prompt each step, and that this is why memory use climbs as a conversation gets longer.
Explain the mechanism: causal masking freezes past hidden states, so keys and values are immutable and append-only, while a query is consumed by its own step. Be ready to state what is stored per layer and per head.
Show you know the operational consequences: the cache is per-session memory that grows linearly with tokens, it is invalidated by edits to earlier turns, and its entries are position-bound because keys carry rotary encoding.
Own the framing that this single storage decision sets the memory economics of serving. Be ready to argue where the field is buying it back — smaller entries per token versus shorter retained history — and what each choice costs the product.
## What autoregressive decoding has to compute A decoder-only transformer emits one token at a time. To produce token t+1 it runs a forward pass whose last position is token t. Inside every attention layer, the layer's input hidden states are projected into three tensors: queries (Q), keys (K) and values (V), each a linear projection of the hidden state at that position. The attention output at position i is a weighted average of value vectors, with weights from the scaled dot product of that position's query against the keys it is allowed to see: softmax(q_i K^T / sqrt(d)) V. ## The causal mask is what makes caching possible Decoder attention is causally masked: position i may attend only to positions 0..i. That single restriction has a strong consequence. The hidden state at position i, in every layer of the stack, is a function of tokens 0..i only. Appending token t+1 therefore cannot change any activation at any earlier position. The key and value vectors computed for position 3 during the very first forward pass are exactly the vectors every later step will need — they are append-only and immutable. That is the ideal shape for a cache: write once, read forever, never invalidate. ## Why the query is the odd one out The query at position i has exactly one consumer: the attention computation that produces position i's own output, at the step when i is the newest token. Nothing at step t > i ever multiplies q_i again — future positions bring their own queries and score them against the accumulated keys. Caching q_i would store a tensor no future step reads. This asymmetry is why the structure is called a KV cache and not a QKV cache, and why a decode step is a matrix-vector operation: a single query row against a tall key matrix. ## What the cache actually holds One key vector and one value vector per KV head, per layer, per position. Not queries, not attention outputs, not MLP activations, not weights. In rotary-position models the key is normally stored with the position encoding already applied, which means a cache entry is bound to the position it occupied — you cannot silently reuse an entry at a different offset without re-encoding it. ## The complexity change Without a cache, generating n tokens means re-encoding a prefix that grows every step, so the total work is a sum over steps of an ever-larger forward pass — quadratic in n before attention's own length-dependent term is counted. With the cache, each step does one token's projections and an attention read over the t existing entries. The saved compute is paid for in memory: bytes scale as tokens x layers x kv_heads x head_dim x 2 (K and V) x bytes per value, and that memory is per concurrent session, not per model. ## It is not an approximation The cache is pure bookkeeping. Mathematically the output is identical to recomputing the prefix; nothing is dropped or summarized. In practice results can differ in the last bits because different batch and sequence shapes change the order of floating-point reductions inside the kernels, but no quality tradeoff is intended or implied. If you see a real quality change when you enable caching, that is a bug or a precision choice, not the mechanism. ## Where the cache is invalid Three cases are worth naming. First, bidirectional models — a masked language model lets every position attend to every other, so appending a token changes all keys and values and nothing is reusable; caching is a decoder-only trick. Second, edited history: if you delete or rewrite an earlier message in a conversation, every cached entry from the edit point onward is stale, because the hidden states downstream of it change. Third, repositioning: because keys carry position information, an entry captured at position 500 is not valid at position 900. ## What to take away The cache exists because causal masking freezes the past. Keys and values are the part of the past that future steps still read, so they are stored; queries are the part that has already done its job, so they are discarded. Everything else about serving long-context models — the memory ceiling, the concurrency limit, the pressure to shrink each entry — follows from that one storage decision.
- Why does a bidirectional encoder gain nothing from this cache?Because it has no causal mask. Every position attends to every other, so appending a token changes the hidden states — and therefore the keys and values — of all existing positions. Nothing computed earlier stays valid, so there is nothing to reuse. KV caching is specific to causally masked, decoder-only generation.
- Are cached keys stored before or after the positional encoding is applied?In standard rotary-position implementations the key is rotated for its position and then cached, so an entry encodes where it sat. That is why a cached prefix is only reusable at the same offset, and why editing an earlier turn invalidates everything after it rather than just shifting entries along.
- Does enabling the cache change what the model outputs?Not mathematically — it computes the same attention over the same keys and values. Small numeric differences can appear because different sequence and batch shapes change floating-point reduction order in the kernels, but there is no approximation, no dropped context and no intended quality tradeoff.
Reading a book, your notes on the pages you have already read stay valid as you read on — but the specific question you asked about page 3 was answered at page 3 and is never asked again.
saying these in an interview costs you the question
- Says the cache stores model weights or every layer's MLP activations
- Claims queries are cached too and simply discarded later
- Thinks caching makes per-step cost independent of context length
- Describes the cache as a lossy approximation of earlier context
- Assumes any transformer, including bidirectional encoders, can cache this way