Continuous Batching

## Why batch at all Generating one token requires a forward pass that streams the model's entire weight matrix out of GPU memory. Those weight reads dominate the cost of a decode step, and they are shared: running eight sequences in the same forward pass reads the weights once, not eight times. Batching is therefore close to free throughput, which is why every serving engine does it. The interesting question is not *whether* to batch but *when the batch membership is decided*. ## Static, request-level batching The straightforward implementation treats a batch like a unit of work. Collect N requests, call a generate loop, and iterate until every sequence in the batch has produced a stop token or hit its length limit. Then return all N responses and start assembling the next batch. This has two separate costs, and candidates usually name only one. The first is **internal waste**. Generation lengths in a real chat workload are heavily skewed: most answers are short, a few are very long. If one sequence runs to 900 tokens and the other seven stop at 30, those seven slots contribute nothing for 870 steps. The GPU still executes a batch of eight positions each step; seven of them are padding or masked no-ops. Utilization looks high on a dashboard while most of the work is fictional. The second is **head-of-line blocking on arrival**. A request that shows up one millisecond after the batch launched cannot be added. Its time-to-first-token now includes the *entire* remaining runtime of a batch it has nothing to do with. Under load this queueing delay, not model compute, becomes the dominant term in what the user experiences. ## Iteration-level scheduling Continuous batching moves the scheduling decision from the request boundary to the **iteration** boundary. The engine keeps two sets: a *running* set that participates in the current forward pass and a *waiting* set of admitted-but-not-yet-scheduled requests. Before each step the scheduler: 1. Removes sequences that finished on the previous step and streams their final token to the client immediately — no waiting for batch-mates. 2. Admits requests from the waiting set into the freed capacity, provided there is a free sequence slot and enough KV-cache space to hold their state. 3. Builds the forward pass over whatever mix of sequences it ended up with. Because each sequence carries its own KV cache, sequences at completely different positions coexist in one batch with no interference: one may be on token 3 of its answer and another on token 800. Nothing is padded to a common length during decode, since each running sequence contributes exactly one new token position per step. A newly admitted request still has to be **prefilled** — its prompt must be run through the model to populate its KV cache before it can decode. That prefill is a large, compute-heavy piece of work relative to a single decode step, which is what makes mixing prefill and decode in one continuously batched server its own tuning problem. ## What continuous batching does *not* do It does not reduce the FLOPs per token, and it does not make an individual request faster in isolation — a single request alone on an idle GPU sees no benefit at all. It is not client-side batching either: the client still sends one ordinary streaming request and knows nothing about who it is batched with. And it is not simply "a bigger batch size": you can run static batching with batch size 256 and still be slower than continuous batching at 64, because the batch-size number describes capacity while continuous batching describes *occupancy*. ## Naming and where you meet it Different projects use different words for the same idea. vLLM's V1 scheduler does iteration-level scheduling by default. TensorRT-LLM calls it **in-flight batching**, and Triton exposes it through the TensorRT-LLM backend — importantly, Triton's generic **dynamic batching** is a *different* mechanism that groups independent inference requests into one larger request-level batch, which is the right tool for a vision model and the wrong tool for autoregressive generation. TGI has done continuous batching since its early versions. ## The knobs that bound it Continuous batching is not unbounded concurrency. Every engine caps how many sequences may run at once — vLLM spells this `max_num_seqs`, while TGI bounds admitted work with `--max-concurrent-requests` — and the real ceiling is usually KV-cache memory rather than the configured cap. When the cache fills, the scheduler stops admitting and may evict already-running sequences, so "the batch is continuous" never means "the queue is empty".