Why does speculative decoding stop paying off as an inference server's batch size grows?

## The regime, stated plainly Speculation is a bet that the GPU has arithmetic capacity going unused during decode. At batch size 1 that bet is overwhelmingly correct: the step's cost is dominated by streaming weights, and adding drafted positions barely moves wall-clock. As the running batch grows, the server is naturally doing more arithmetic per weight read, and the unused capacity disappears. Past that point every drafted position is real work, and the roughly (k+1)x multiplier on positions-per-sequence turns into a roughly proportional cost. That is the core of the answer, but the interviewer usually wants the serving-specific consequences, which are more interesting than the regime argument itself. ## Effect 1: rejected work is stolen from other requests At low load, wasted verification FLOPs cost nobody anything — the alternative was idle silicon. At high load the GPU is the shared resource every queued request is waiting for, so a rejected drafted token is capacity taken from a real user. A 60%-acceptance setup at draft length 4 discards a large share of its verification work; at low load that is invisible, at saturation it is a direct throughput tax. This is why aggregate tokens per second on a busy fleet usually *falls* when speculation is enabled, even though per-request latency looks better in a single-stream test. ## Effect 2: batches do not accept uniformly Every sequence in a batch drafts its own candidates and accepts a different number of them. The iteration must evaluate all drafted positions for all sequences, so the work is set by the full draft width, while the useful output is set by each sequence's own acceptance. Some engines mitigate this by dropping speculation for sequences whose recent acceptance is poor, but the general shape holds: heterogeneous acceptance inside a batch means padding waste that grows with batch size. ## Effect 3: the memory bill hits concurrency A draft model occupies weights and, for every in-flight sequence, its own KV cache. Both come out of the same GPU memory that would otherwise hold request KV. Fewer concurrent sequences fit, so under load requests spend longer in the queue. It is entirely possible to enable speculation, improve inter-token latency, and *worsen* end-to-end p95 because queueing grew. When someone reports that, the first thing to compare is admitted concurrency before and after. ## Effect 4: draft steps serialise With a draft model, the k small forward passes happen before verification and are on the critical path for the whole batch. As batch size grows those draft passes also grow in cost, so the overhead does not stay constant — it scales with the batch you were hoping to amortise it over. ## What to do about it **Gate on load.** The cleanest answer is dynamic: skip speculation when the running batch exceeds a threshold, and use it when the server is quiet. This is a first-class feature in serving engines — vLLM's speculative configuration, for example, exposes a batch-size threshold above which speculation is bypassed. Set the threshold empirically at the point your measured speedup crosses 1.0. **Split the fleet.** If you have both interactive and bulk traffic, run two pools: a speculation-enabled pool sized for low concurrency and strict latency, and a plain pool tuned for batch throughput. Routing by workload is usually cleaner than trying to make one pool serve both, because the two want opposite scheduler settings. **Measure goodput, not speedup.** The number that decides this is how many requests per second you serve *within* your latency target, at production concurrency. A single-stream speedup measurement will always flatter speculation, because single-stream is exactly the regime it is designed for. **Re-check after any capacity change.** Adding replicas, changing GPU SKU, or a traffic shift all move where the crossover sits. The threshold is a measured property of a specific deployment, not a constant. ## The one-line version Speculation buys latency with spare compute. When there is no spare compute — because your users are already using it — there is nothing left to buy it with, and continuing to speculate makes the fleet slower for everyone.