Why does prompt caching cut time-to-first-token but not tokens-per-second?
answer
- two phases, only one is skippable
- one parallel pass versus many sequential steps
- output length sets the step count
- decode is bound by memory bandwidth
basics
~20 sCaching removes prefill — the single parallel pass that builds key/value state for the prompt. Decoding still runs one sequential forward pass per generated token, and those steps are untouched, so the answer starts sooner but streams at the same rate.
solid answer
~50 sInference has two phases with completely different profiles. **Prefill** processes the whole prompt in one pass: all 50,000 tokens go through the model together, as large matrix multiplications, so it is compute-bound and highly parallel — and it is the only thing standing between the request arriving and the first token appearing. **Decode** then produces output one token at a time; 500 output tokens means 500 strictly sequential passes, each reading the full weights and the whole KV cache from memory to do one token of work, which makes it memory-bandwidth-bound. A cache hit hands back the prefix's key/value tensors, so prefill over that span disappears — that is a pure reduction in time-to-first-token. It does nothing to the 500 decode steps, whose count is set by output length and whose speed is set by memory bandwidth. If anything a longer cached prefix makes each decode step marginally slower, because attention reads more cached state per step.
go deeper
Remember that caching helps the model start answering sooner, because it skips work on the prompt, and that the speed at which words then stream out is a separate thing.
Name the two phases and what each costs: prefill is one parallel pass over the prompt, decode is one sequential pass per output token, and only the first is skippable.
Demonstrate the diagnosis — measure time-to-first-token and inter-token latency separately, expect only the former to move, and be able to say which workload shapes make caching worth designing around at all.
Own the shape argument: caching is a lever on long-context, short-output traffic and nearly irrelevant on generation-heavy traffic, so the decision to build a caching strategy should follow from the traffic mix rather than from the feature existing.
## Two phases, not one Every LLM request runs in two distinct regimes, and confusing them is the root of most latency misdiagnosis. **Prefill** ingests the prompt. All prompt tokens are pushed through the model in a single forward pass, batched along the sequence dimension. The work is dominated by big dense matrix multiplications with high arithmetic intensity, so it saturates the GPU's compute units. Its output is the key/value state for every prompt position, plus the logits for the final position — the first generated token. Prefill time grows with prompt length, roughly linearly at moderate lengths and worse once attention's quadratic term matters. **Decode** produces the answer. Each step feeds exactly one token through the model, attends over the accumulated KV cache, samples the next token, and repeats. The steps are strictly sequential — step *n+1* cannot begin before step *n* has chosen a token. Each step performs very little arithmetic but must stream the entire weight matrix and the whole KV cache out of memory, so it is bandwidth-bound, and the achievable tokens per second is essentially a property of memory bandwidth and model size, not of prompt length. ## Where a cache hit lands A prompt cache stores exactly the artifact prefill produces: the per-layer keys and values for a span of prompt tokens. On a hit the server adopts that state and runs prefill only over the tokens that follow the matched prefix. Consider a 50,000-token prompt with a 500-token answer. On a miss you pay one 50,000-token parallel prefill pass and then 500 sequential decode steps. On a full-prefix hit you pay a near-zero prefill and then the same 500 sequential decode steps. The phase that was removed is entirely upstream of the first token, so the metric that moves is **time-to-first-token**. The phase that dominates wall-clock for long outputs is untouched, so **inter-token latency and tokens-per-second stay flat**. ## Why decode cannot be cached away Decode is not repeated work — each step produces a token that did not exist before and that the next step depends on. There is nothing to look up. The only KV reuse inside decode is the ordinary within-request kind that every server already does. This is also why the benefit profile is so lopsided by workload shape: a retrieval-augmented request with a 40,000-token context and a 200-token answer is dominated by prefill and transformed by caching; a request with a 1,500-token prompt and a 4,000-token answer is dominated by decode and barely notices. ## The second-order effect people miss A long cached prefix does not speed decoding — it slightly slows it. Every decode step's attention must read keys and values for all preceding positions, so a 50,000-token shared prefix means 50,000 positions' worth of cached state streamed from memory on every one of the 500 steps. The tokens are free to *build* on a hit, but never free to *attend over*. Candidates who claim caching "makes the whole request faster proportionally" have not internalised this. ## How you would confirm it Separate the two measurements rather than reporting one end-to-end number. Time-to-first-token isolates prefill (plus queueing and any cache load); the mean gap between streamed tokens isolates decode. A working cache shows a large drop in the first and a flat line in the second. If both move together, something else changed — a different model, a different batch, a routing change — and the cache is not what you are observing. ## The cost that is not zero A hit is cheaper than prefill but not free. The tensors must be resident in the attention path; if the server spilled them to host memory or disk, they have to be transferred back, and that transfer time shows up inside time-to-first-token. For very long prefixes this is still far cheaper than recomputation; for short ones the transfer can approach the cost of just recomputing, which is one reason serving stacks apply a minimum useful match length. ## The one-line version Caching skips the parallel pass that happens once; it cannot skip the sequential passes that happen once per output token. Judge it by time-to-first-token, and expect streaming speed to be unchanged.
- For a workload with a 2,000-token prompt and a 4,000-token answer, how much does prefix caching help?Very little. Prefill over 2,000 tokens is a small share of the wall clock next to 4,000 sequential decode steps, so removing it shaves a modest slice off time-to-first-token and leaves total latency roughly where it was. Caching pays on long-context, short-output shapes; this is the opposite shape.
- Does a longer cached prefix make decoding faster?No — marginally slower. Every decode step attends over all preceding positions, so a longer prefix means more cached keys and values streamed from memory per step. What caching buys is the one-off construction of that state, never the per-step cost of reading it.
- Why is decode described as memory-bandwidth-bound while prefill is compute-bound?Prefill multiplies a whole sequence of tokens against the weights at once, so each byte of weights loaded does a lot of arithmetic. Decode does the same weight loads for a single token, so arithmetic intensity is tiny and the step time is set by how fast weights and KV can be streamed out of memory.
- Is a cache hit always faster than recomputing the prefix?Almost always, but not by definition. If the entries were spilled to host RAM or disk, they must be transferred back, and that transfer sits inside time-to-first-token. For long prefixes the transfer is far cheaper than the prefill FLOPs; for very short ones the two can converge, which is why serving stacks ignore trivially short matches.
saying these in an interview costs you the question
- Claims caching speeds up generation, not just the first token
- Treats prefill and decode as one undifferentiated phase
- Thinks decode steps can be looked up or reused across requests
- Assumes a long cached prefix makes each decode step cheaper
- Judges cache effectiveness by end-to-end latency on long-output workloads