skip to content

When does automatic prefix caching cut TTFT, and when does it silently miss?

level: middleimportance: must knowfreq 58%

answer

  1. skips prefill, not decode
  2. exact token match from position zero
  3. chained hash per complete block
  4. variable content belongs at the end
  5. running sequences outrank cached prefixes

basics

~20 s

Prefix caching reuses already-computed KV blocks when a new request starts with the exact same tokens as an earlier one, skipping that part of prefill. It misses whenever the prompt differs at the front, when the shared run is shorter than one block, or when those blocks have already been evicted.

solid answer

~60 s

The engine hashes each *complete* block of prompt tokens together with the hashes of all preceding blocks, so a hash identifies a whole prefix, not a fragment. On a new request it walks the prompt block by block, and every block whose chained hash is already in the cache is reused by reference — its prefill compute is skipped entirely. That is a direct time-to-first-token win on workloads with a long shared system prompt, few-shot examples, a re-sent conversation history, or repeated queries over the same document. It misses in four ways, all of them quiet. Matching is a *prefix* match, so anything that varies at the front — a timestamp, a user id, a shuffled instruction block — invalidates everything after it. Only full blocks are cacheable, so a shared run shorter than the block size buys nothing and the trailing partial block is always recomputed. Cached blocks with no live reference are evicted, typically in least-recently-used order, when the memory is needed for running sequences. And it saves prefill only — a request that shares a prefix still decodes at the same speed.

code

python · 15 lines
python
BLOCK = 16

def reusable_blocks(prompt_a, prompt_b):
    n = min(len(prompt_a), len(prompt_b)) // BLOCK
    hit = 0
    for i in range(n):
        s = slice(i * BLOCK, (i + 1) * BLOCK)
        if prompt_a[s] != prompt_b[s]:
            break
        hit += 1
    return hit

shared = list(range(1000))
print(reusable_blocks(shared + [1], shared + [2]))        # 62 blocks reused
print(reusable_blocks([999] + shared, shared + [2]))      # 0: differs at token 0

go deeper

for a junior

Know that if a new request begins with exactly the same tokens as an earlier one, the server can reuse that work and answer its first token sooner.

for a middle

Be able to explain block-aligned exact-prefix matching and name the four misses: front-loaded variation, sub-block prefixes, eviction under pressure, and no effect on decode.

for a senior

Show you would design the prompt for it — fixed content first, variable last — and that you would watch hit rate next to cache utilization, reading a falling hit rate at high utilization as thrash rather than a misconfiguration.

for a principal

Own the economics and the isolation question together: reuse is what makes a long shared system prompt affordable at scale, and a cache shared across tenants is a design decision that must account for what distinguishes them.

## The mechanism A paged KV cache already stores each sequence's keys and values in fixed-size blocks, and a physical block can be referenced by more than one sequence. Prefix caching is what turns that capability into automatic reuse across *unrelated* requests. When a block of prompt tokens is filled, the engine computes a hash over the block's token ids **chained with the hash of the previous block**. The chaining is what makes the key mean "this exact sequence of tokens from position zero" rather than "these sixteen tokens somewhere". Additional inputs are folded in where they would change the computed state — a LoRA adapter identity, multimodal input hashes — so two requests that agree on text but differ in adapter do not collide. The engine keeps a map from chained hash to physical block. When a request arrives, it hashes the prompt block by block and looks each up. Every hit is claimed by incrementing that block's reference count and writing its index into the new sequence's block table. The first miss ends the walk: everything from there on must be computed. So the reused portion is always a contiguous prefix, never a middle fragment. ## Why it shows up as TTFT Time-to-first-token is dominated by prefill — the one forward pass over the whole prompt. Reusing k blocks removes those tokens from the prefill entirely, so TTFT falls roughly in proportion to the reused fraction, and the GPU work saved becomes capacity for other requests. On a chat API with a 1,500-token system prompt and 200-token user turns, a warm cache can remove most of the prefill on nearly every request. That is the single largest cheap win available in this part of the stack, which is why vLLM enables `enable_prefix_caching` by default in its V1 engine, TensorRT-LLM offers `enable_block_reuse` in its KV-cache configuration, and recent TGI releases turn prefix caching on without configuration. ## The four silent misses **Front-loaded variation.** Put the current timestamp or the user's name at the top of the system prompt and the very first block's hash changes, so nothing after it can match. The fix is layout: fixed content first, variable content last. This is the most common and most invisible failure — nothing errors, the server is just slower and more expensive than it should be. **Sub-block prefixes.** Only complete blocks are hashed. With a 16-token block, a 12-token shared preamble is not reusable at all, and a 300-token shared preamble reuses 288 tokens while recomputing the remaining 12 in the request's own partial block. **Eviction.** Cached blocks that no live sequence references are reclaimable. When running sequences need blocks, the engine evicts these in least-recently-used order. Under sustained pressure the cache thrashes and hit rate collapses exactly when load is highest — the opposite of when you want it. Capacity dedicated to *running* sequences always wins over capacity holding a speculative prefix. **Decode is untouched.** A perfectly matched 8,000-token prefix does nothing for inter-token latency; the request still generates one token per step at the same speed. Prefix caching improves TTFT and throughput, not per-token latency. ## Operating it The metric that matters is hit rate, and engines expose prefix-cache hit and query counters for exactly this. A hit rate far below what your prompt structure predicts points at one of the four misses above — usually the first. Watch it alongside cache utilization: a high utilization with a falling hit rate is the thrash signature. Correctness is worth stating clearly because interviewers probe it: reuse is exact-match on token ids, not semantic. The cached keys and values are precisely what the model would have computed for those tokens in those positions, so output is unchanged. It is a pure compute saving, not an approximation. The one genuine caution is isolation — cached blocks are shared across requests and therefore across tenants, so the hash must incorporate everything that distinguishes them, and a shared cache is a side channel a careful design accounts for when tenants must not learn what other tenants asked. ## Distinguish it from routing Prefix caching lives inside one engine instance. Getting a follow-up request *to the replica whose cache is already warm* is a separate, load-balancer-level concern. A perfect prefix cache with round-robin routing across eight replicas has a one-in-eight chance of landing warm.

  • A tenant id is injected at the top of every system prompt and hit rate is near zero. What do you change?
    Move the variable content to the end of the prompt so the fixed preamble occupies whole blocks from position zero. Matching is a prefix match on token ids, so anything that differs at the front invalidates every block after it. If per-tenant text must lead for policy reasons, accept that each tenant gets its own cache lineage and size the cache for the number of distinct tenants rather than expecting cross-tenant reuse.
  • Does prefix caching change what the model outputs?
    No. The reused blocks contain exactly the keys and values the model would have recomputed for those token ids at those positions, so the forward pass sees identical state. It is a compute saving, not an approximation. That is why engines can enable it by default — the only real caution is isolation, since blocks are shared across requests, which is why adapter identity and multimodal inputs are folded into the hash.
  • Under heavy load your hit rate collapses. Why, and is that a bug?
    Not a bug — it is the intended priority. Blocks holding a cached prefix with no live sequence referencing them are reclaimable, and the scheduler evicts them, usually least-recently-used first, so running sequences can allocate. Under sustained pressure the cache thrashes. The response is capacity or admission control, not disabling eviction, because starving running requests to protect a speculative cache would be strictly worse.
  • Why doesn't prefix caching help a request that shares its prefix but generates a very long answer?
    Because it only removes prefill work. The shared prompt's keys and values are reused, so TTFT drops, but every generated token still costs one full decode step through all layers. For a request producing thousands of tokens, prefill is a small share of end-to-end time, so the visible improvement is limited to the first token — while the GPU work you freed still helps overall throughput.

saying these in an interview costs you the question

  • Thinks it does semantic or fuzzy matching of similar prompts
  • Claims it speeds up token generation, not just prefill
  • Assumes a shared substring anywhere in the prompt is reused
  • Believes cached blocks survive regardless of memory pressure
  • Confuses it with routing a user back to the same replica

context

resolving links…