In vLLM, what does automatic prefix caching hash to match a reused prefix?

## Why hashing at all The KV entries for a token depend on that token *and every token before it*. Two requests can therefore share KV state only when they share an exact prefix. vLLM's job is to recognize that cheaply, at block granularity, without comparing whole prompts token by token against every cached sequence. The answer is a content-addressed block pool: give each block an identifier derived from its contents *and* its position in the sequence, and look that identifier up. ## The chained key For the first block, the key is derived from its token ids. For every block after it, the key combines the **previous block's hash** with **this block's token ids**. The result is a chain: block 5's identity encodes tokens 0 through 5x`block_size`, not just its own 16 tokens. That chaining is the part worth explaining, because the naive alternative — hash each block's tokens alone — is broken. The same 16 tokens can occur at different positions in different prompts, preceded by entirely different context, and their KV values would be completely different. Hashing tokens alone would happily hand you another request's numbers. The chain pins each block to its entire preceding prefix, so a hit is only granted when the KV values genuinely would be identical. The direct consequence is that matching is **prefix-only**. Two prompts sharing a 2,000-token suffix but differing in their first token share nothing: the first block's hash differs, so every downstream hash differs. ## Extra keys Token ids are not the only thing that determines KV values. vLLM folds additional keys into the hash for anything else that does: - **LoRA adapter id** — a different adapter changes the projections, so the same tokens under a different adapter must not collide onto the same block. - **Multimodal input hashes** — an image placeholder token is meaningless without knowing which image it stands for. This is a correctness mechanism, not an optimization. Without it, prefix caching would silently return another tenant's activations for the same visible token sequence. ## Full blocks only A block is hashed and made reusable only once it is complete. A 100-token prompt with a block size of 16 yields six full blocks (96 tokens) plus a partial block holding the last 4, and only the six are cacheable. Practical effect: reuse is quantized to the block size, so of a shared 1,000-token system prompt you reuse 992 tokens, not 1,000. Nobody cares about the remainder — but it explains why cached-token counts never quite equal the prefix length. ## Hits, sharing, and eviction On a hit, the engine does not copy anything. It maps the existing blocks into the new request's block table and increments their reference counts, then prefills only the tokens after the matched prefix. So a hit saves *compute* (skipped prefill, hence lower TTFT for long shared prefixes) and simultaneously saves *memory*, because a hundred requests on one system prompt hold one physical copy of its KV state rather than a hundred. Cached blocks whose reference count drops to zero are not freed immediately; they stay in the pool as reuse candidates until the pool needs them for a live allocation. The eviction policy that governs that, and what happens more broadly when free blocks run out, belongs to the paged-KV-cache topic — here the relevant point is that caching costs nothing extra in memory, because it only retains blocks that would otherwise sit free. ## Defaults and observation In vLLM 0.27 the V1 engine enables prefix caching by default; you turn it *off* with `--no-enable-prefix-caching` (the auto-generated negation of the boolean flag). Turning it off is occasionally right — a workload with no shared prefixes at all pays a small hashing cost for zero hits, and disabling it removes a variable when benchmarking cold prefill. To see whether it is earning its keep, compare `vllm:prompt_tokens_cached` with `vllm:prompt_tokens`: the ratio is the share of prompt tokens that never had to be prefilled. A ratio near zero on a workload you believed had a big shared system prompt is a signal to look at what varies early in your prompts — a timestamp, a session id, a shuffled tool list at the top of a system message will each kill every hit downstream of it. ## The misconception to avoid Prefix caching is not response caching. It does not remember what the model *said*; it remembers the intermediate attention state for a token prefix, and generation proceeds normally from there. Two identical prompts with a non-zero temperature will still produce different outputs — they just get to the first sampled token faster.