When would you split prefill and decode onto separate LLM server instances?

## The tension this resolves A continuously batched server runs two workloads that want opposite things. **Prefill** is a compute-bound burst: thousands of token positions through the model at once, saturating the math units, arriving unpredictably with whatever prompt sizes users send. **Decode** is a memory-bandwidth-bound trickle: one position per sequence per step, dominated by streaming weights and KV cache out of memory, and it wants many concurrent sequences and short, predictable step times. Colocating them on one GPU forces one set of scheduler budgets to serve both. Chunked prefill makes the compromise much better than it used to be — it slices prefill so decodes are not stalled behind whole prompts — but it is still a compromise. A fat token budget favours ingestion and first-token latency; a thin one favours stream smoothness. You get one dial for two contracts, and at scale the two contracts diverge. ## What disaggregation does Split the deployment into a **prefill pool** and a **decode pool**. A request goes to a prefill worker, which runs the prompt through the model and produces its KV cache. That cache is then transferred to a decode worker, which owns the sequence for the rest of its life and generates tokens with no prefill work ever competing for its steps. The consequences follow directly and this is the shape of the argument to make out loud: - **Independent sizing.** Prefill capacity scales with input tokens per second; decode capacity scales with concurrent sequences and output tokens per second. In a colocated fleet you must buy both in the ratio your model happens to impose. Disaggregated, you buy each to fit the actual traffic — which matters enormously because the prompt-to-output ratio varies by an order of magnitude between a summarization service and a chat product. - **Independent tuning.** Prefill workers can use a huge token budget with no stream to protect. Decode workers can run high concurrency with short, uniform steps. Neither compromises for the other. - **Clean SLOs.** First-token latency becomes a property of the prefill pool plus the transfer; per-token latency becomes a property of the decode pool. Each has one owner and one scaling signal, instead of a shared queue where a spike in long prompts silently degrades everyone's token rate. - **Different hardware, potentially.** Prefill wants math throughput; decode wants memory bandwidth and capacity. At sufficient scale that is an argument for different GPU SKUs per pool. ## What it costs The KV cache must physically move. For a long prompt on a large model that is gigabytes per request, and it has to travel over whatever interconnect exists between the pools — fast within a node, much slower across nodes. That transfer sits directly in the first-token latency path, so a disaggregated deployment on a weak interconnect can be *worse* on TTFT than a colocated one, which is the failure mode to name. Beyond bandwidth: you now operate two services with two scaling policies, a routing layer that must track which decode worker owns which sequence, and a failure model where losing a decode worker mid-generation loses in-flight sequences that a colocated replica would have kept. Prefix reuse also gets harder — a shared system prompt cached on a prefill worker does not automatically help a decode worker, so cache-locality routing becomes its own design problem. ## When to say yes, and when to say no Say yes when: the fleet is large enough that both pools are meaningfully sized; the prompt-to-output ratio is lopsided enough that colocated sizing wastes real money; you have separate, contractual first-token and per-token targets; and your interconnect is fast enough that the transfer is a small fraction of your TTFT budget. Say no when: you run one or two replicas, where the operational complexity dwarfs any gain; the interconnect is ordinary networking and the transfer would dominate; prompts are short, so prefill is cheap and never really interferes; or you have not yet tuned the colocated case, because chunked prefill plus sensible scheduler budgets recovers a large share of the benefit for none of the complexity. ## Where it stands as technology This is an actively moving area rather than settled practice. vLLM supports prefill/decode disaggregation through a KV connector layer that handles transferring cache between instances, and the large-scale serving frameworks built on top of these engines have made it a headline feature. Treat it as a scale play whose economics you should be able to justify with the ratio of input to output tokens in your own traffic — the interviewer is testing whether you reach for architecture before you have exhausted configuration, not whether you can recite the design.