Why is LLM prefill compute-bound while token-by-token decode is bandwidth-bound?

## The two phases A request is served in two structurally different phases. **Prefill** runs the entire prompt through the model once, producing the KV cache for every prompt position and the first output token. **Decode** then runs one forward pass per generated token, each pass appending a single entry to the cache. Both phases execute exactly the same layers and the same math; what differs is the shape of the operands, and that shape decides which piece of hardware runs out first. ## Arithmetic intensity is the whole story Roofline analysis asks a single question of a kernel: how many FLOPs does it perform per byte it reads from memory? An accelerator has a peak math rate and a peak memory bandwidth; their ratio — typically in the hundreds of FLOPs per byte on current data-center parts — is the break-even intensity. Below it you are bandwidth-bound, above it compute-bound. In prefill, a projection is a matrix-matrix multiply: a weight matrix of d x d is read once and multiplied against P token vectors. The bytes read stay proportional to d^2 while the FLOPs scale with P x d^2, so intensity rises with prompt length. A few hundred tokens is already enough to sit comfortably on the compute side of the roofline. In decode, P is 1. The same weight matrix is read in full and each element participates in about two FLOPs. Intensity is roughly one to two FLOPs per byte — one to two orders of magnitude below break-even. The tensor cores spend most of their time waiting for weights to arrive. On top of that, attention in decode must read the entire KV cache for the sequence, and that read grows with context length, so long sessions move even more bytes per generated token. ## What each phase's cost scales with Prefill cost scales with prompt length for the projections and feed-forward blocks, and with the square of prompt length for full attention scores — which is why long-prompt systems reach for sparse or linear attention variants and for reusing previously computed prefixes. Decode cost per step is roughly constant in weight FLOPs but grows in bytes read as the cache lengthens, so inter-token latency drifts upward across a long session even though the model is doing the same amount of arithmetic each step. ## Why this matters for tuning Because the bottlenecks differ, so do the remedies, and applying the wrong one wastes money. A prefill-heavy service — long documents, short answers — benefits from raw FLOPs, better attention kernels, cheaper attention math and avoiding re-encoding of content the system has already processed. A decode-heavy service — short prompts, long generations — barely notices extra FLOPs; it improves when you reduce bytes moved per token: quantizing weights, quantizing or shrinking the KV cache, choosing architectures with smaller per-token cache entries, or having more sequences share each weight sweep so a single weight read serves many tokens at once. ## A concrete split Consider an IDE inline-completion service. Each keystroke sends a 6,000-token file prefix and the model returns a 20-token completion. The prefill is one parallel pass over 6,000 tokens — a large slab of arithmetic that the hardware is well-shaped to do. The decode is 20 sequential passes, each re-reading the full weight set and the 6,020-entry cache to emit one token. The prefill does far more total arithmetic; the decode does far more sequential trips to memory. Which one dominates the wall clock depends on model size, prompt length and hardware, so measure rather than assume — but they will never respond to the same optimization. ## Telling which regime you are in Instrument, do not guess. During prefill you should see high tensor-core utilization and FLOPs near a healthy fraction of peak. During decode you should see memory throughput near peak with math utilization low. If decode shows neither, you are limited by something else — kernel launch overhead, small batches, host-side scheduling or synchronization — and adding hardware will not help. ## The serving consequence Because the two phases want different things, production stacks increasingly run them separately: a long prefill sharing a device with active decodes stalls those decodes and spikes their inter-token latency, so engines either chop prefill into chunks that interleave with decode steps, or place the phases on different pools of hardware and transfer the cache between them. The lesson at the method level is the same either way: prefill and decode are two workloads wearing one model, and treating them as one is how capacity plans go wrong.