skip to content

In a paged KV cache, what does raising the block size gain and cost?

level: seniorimportance: should knowfreq 38%

answer

  1. granularity, on both sides
  2. fewer table lookups, longer reads
  3. waste bounded by block minus one
  4. only whole blocks are reusable
  5. the default is backend-tuned

basics

~20 s

Bigger blocks mean fewer table entries and longer contiguous reads, so kernels and bookkeeping get cheaper. The cost is coarser granularity: more wasted space in each sequence's partly-filled tail block, and prefix sharing that only matches in larger chunks.

solid answer

~50 s

Block size is the token span one physical block covers — vLLM exposes it as `block_size`, commonly 16. Raising it makes each attention-kernel iteration read a longer contiguous run, which coalesces better and cuts the number of dependent block-table loads; it also shrinks per-sequence bookkeeping, since a 32k sequence needs 2,000 table entries at 16 tokens but 250 at 128. The costs are all granularity. Internal fragmentation is bounded by the block size minus one token per sequence, so wide blocks strand real memory once you have hundreds of concurrent sequences. Prefix caching only reuses *full* blocks, so a large block means a shared prefix must match in longer chunks before any of it is reusable, and the trailing partial chunk is always recomputed. Copy-on-write also copies more per divergence. In practice the default is tuned per attention backend and is rarely the right thing to move first; measure before touching it.

code

python · 7 lines
python
def blocks_and_waste(seq_len, block_size, concurrent):
    blocks = -(-seq_len // block_size)          # ceil
    waste = (blocks * block_size - seq_len) * concurrent
    return blocks, waste

for bs in (16, 64, 256):
    print(bs, blocks_and_waste(seq_len=1000, block_size=bs, concurrent=256))

go deeper

for a junior

Know that the paged cache is divided into fixed-size blocks and that this size is a configurable knob, not something the model itself defines.

for a middle

Be able to state both directions: fewer, longer reads and smaller tables as blocks grow, against more stranded space in each sequence's tail block.

for a senior

Show the second-order effect interviewers are actually after — coarse blocks lower prefix-cache hit rate and enlarge copy-on-write copies, so a kernel-level win can be a workload-level loss. Say that you would measure hit rate and tokens per second, not the knob.

for a principal

Own the judgment that per-backend defaults encode kernel constraints, so this knob is near the bottom of the tuning list; the levers that move cost per token are the memory budget, concurrency limits and prefix reuse.

## What the knob is A paged KV cache is carved into fixed-size physical blocks, each holding the keys and values for a fixed number of consecutive token positions. That number is the block size. vLLM surfaces it directly as the `block_size` engine argument and picks a platform-appropriate default — 16 tokens is the common value on CUDA, and some attention backends require or prefer other values. TensorRT-LLM has the equivalent notion in its KV-cache configuration. The point of the question is not the default; it is that this single number sits on both sides of a real tradeoff. ## What gets better as blocks grow **Kernel efficiency.** A paged attention kernel processes the cache block by block. Every block boundary means resolving a block-table entry — a dependent load — before the next chunk of keys and values can be fetched. Larger blocks mean fewer boundaries, longer uninterrupted reads, better memory coalescing, and a shape closer to what a contiguous kernel would have enjoyed. **Bookkeeping.** Block tables, reference counts and free-list operations all scale with block *count*, not with tokens. Doubling the block size halves the tables the host must maintain and ship to the device each step. At very long context and high concurrency this is not nothing. **Allocation frequency.** A sequence requests a new block only when its tail block fills. With 128-token blocks that happens eight times less often than with 16, so the scheduler's allocation path runs less. ## What gets worse **Internal fragmentation.** Every sequence has exactly one partially-filled tail block, and the space after its last token is unusable by anyone else. The bound is the block size minus one token per sequence. With 16-token blocks and 256 concurrent sequences that is under 4,000 stranded token-slots — noise. With 256-token blocks it is up to 65,000 token-slots, which at long-context KV sizes is real gigabytes. **Prefix-cache granularity.** Automatic prefix caching hashes and reuses only *complete* blocks. If two requests share 300 tokens of system prompt, a 16-token block size lets 18 blocks (288 tokens) be reused and recomputes the remaining 12 tokens; a 256-token block size reuses one block and recomputes 44 tokens. Worse, a shared prefix shorter than one block is entirely unreusable. Coarse blocks quietly lower your cache hit rate on exactly the workloads prefix caching exists to serve. **Copy-on-write cost.** When two sequences sharing a block diverge, the engine copies the whole block. Bigger block, bigger copy, on every divergence — which matters for parallel sampling and for any workload that forks from a common prompt. **Scheduling coarseness.** Preemption and admission decisions are made in units of blocks, so a coarse block size makes the scheduler's memory accounting lumpier and its decisions less precise near saturation. ## How to think about choosing Start by not choosing: the engine's per-backend default exists because the attention kernel it dispatches to has an opinion, and some backends only support specific values. Overriding it can silently drop you onto a slower kernel path — a common self-inflicted regression. If you do tune it, the workload tells you the direction. Long sequences, low concurrency, and little prefix sharing tolerate larger blocks and benefit from the kernel and bookkeeping savings. Short prompts, very high concurrency, and heavy prefix reuse — the typical chat-API shape — want small blocks, because fragmentation is multiplied by sequence count and cache hit rate is multiplied by request rate. And measure the thing you care about, not the knob: cache utilization at a fixed load, prefix-cache hit rate, tokens per second at your latency target. A block-size change that improves kernel microbenchmarks while cutting hit rate is a net loss on a workload with a shared system prompt. ## The common wrong answers Block size is not batch size, not the maximum model length, and not a quality knob — attention output does not depend on it. It is also not a way to "fit a longer context": total cache capacity is set by the memory reserved for it, not by how that memory is subdivided. Candidates who reach for block size as their first tuning lever are usually solving a problem that belongs to concurrency limits or to the memory budget.

  • Why is a shared prefix of 200 tokens partly recomputed even when prefix caching is on?
    Because only full blocks are hashed and reused. With a 16-token block, 192 of those tokens fall into 12 complete blocks that can be reused, and the trailing 8 tokens live in a partial block that is not cacheable, so they are recomputed as part of the new request's prefill. The larger the block, the larger that recomputed remainder can be.
  • Would a block size of 1 give perfect memory efficiency?
    It would eliminate internal fragmentation, but it is a bad trade. Every token would need its own table entry and its own dependent lookup, so the block table becomes as large as the sequence, host-to-device bookkeeping explodes, and the attention kernel degenerates into a fully scattered gather with no coalescing. The gains from paging come from indirection being cheap, and per-token indirection is not.
  • You raised the block size and throughput fell on a chat workload. What is the first thing to check?
    Prefix-cache hit rate. Chat traffic shares a long system prompt, and coarser blocks mean a shorter reusable run plus a bigger recomputed remainder, so prefill work per request goes up even though the kernel got marginally faster. Check whether the change also pushed you onto a different attention backend, since some kernels only support particular block sizes.

saying these in an interview costs you the question

  • Treats block size as a way to extend the context window
  • Says larger blocks always improve throughput
  • Ignores that only complete blocks are reusable by prefix caching
  • Confuses block size with the concurrent-sequence limit or batch size
  • Assumes block size affects output quality

context

resolving links…