What does TGI's --speculate flag enable, and when does it stop paying off?

## The flag `--speculate` takes a number: how many tokens to propose beyond the one the model would normally produce this step. It can also be supplied through the launcher's environment variable form, and it is a server-level setting - you choose it when you start the container, not per request. What provides the proposals depends on what you are serving: - **Medusa-style checkpoints** ship extra prediction heads trained on top of the base model. Each head guesses a token at a future offset, so `--speculate` is effectively how many heads you use. - **Any other checkpoint** falls back to **n-gram speculation**: TGI looks for the recent token sequence elsewhere in the request's own context and proposes whatever followed it there. No draft model, no extra weights, no extra VRAM. The base model then runs one forward pass that both verifies the proposed tokens and produces the next one. Accepted proposals commit immediately; the first rejection truncates the rest. ## Why it helps - and why the help is conditional Single-stream decode is bandwidth-bound: each step streams the model weights through the GPU to produce one token, leaving arithmetic units largely idle. Verifying four candidate tokens in that same pass costs almost nothing extra in memory traffic, so if the guesses are right you got several tokens for one weight read. That is the entire trick - it converts idle compute into fewer sequential steps. The condition is that the compute was actually idle. As concurrency rises and the running batch grows, each forward pass already has plenty of arithmetic to do; the weights are being amortised across many sequences and the GPU is no longer waiting on memory. Now the speculative verification is real added work competing with real requests, and net throughput can fall below the unspeculated baseline. This is why speculation is a latency feature for lightly-loaded or interactive deployments rather than a throughput feature for saturated ones - and why you must benchmark it at your *actual* load, not at concurrency one. ## Where n-gram speculation earns its keep n-gram speculation has no draft model, so its guesses are only good when the output genuinely echoes the input. That is common in exactly the workloads people self-host for: - **Code editing / refactoring** - most of the emitted file is copied from the file you pasted. - **RAG answering and summarisation** - phrases and names are lifted from retrieved passages. - **Structured rewrites and translations of templated text** - large spans are verbatim. On free-form creative generation, where nothing in the prompt predicts the continuation, proposals are rejected constantly and you have paid verification cost for nothing. ## Choosing N Bigger N is not monotonically better. Each additional proposed token costs verification work whether or not it is accepted, and the probability that the whole run of guesses survives falls with length. Small values are the usual sweet spot; the honest method is to sweep N against your own workload and watch end-to-end latency and tokens per second together, not just one of them. Watch two things while you sweep: inter-token latency (should improve if speculation is helping) and total throughput at your target concurrency (must not regress). If they move in opposite directions, you are seeing the compute-contention effect and should either lower N or turn speculation off for that deployment. ## Operational cautions - Medusa requires a checkpoint that actually has the heads. Passing `--speculate` at a plain checkpoint gets you the n-gram path, not Medusa. - Speculation changes the shape of each forward pass, which interacts with the memory budget the server profiled at startup - re-check headroom after enabling it rather than assuming the previous batch sizing still holds. - It is a server-wide switch. If one route on the endpoint is code-completion (great acceptance) and another is chat (poor acceptance), one setting serves both; that is an argument for separate deployments. ## What interviewers listen for A weak answer says 'it makes generation faster'. A strong one names the resource being traded (idle compute for fewer sequential steps), names the regime where the trade stops working (a saturated batch), and names the workloads where n-gram proposals actually land. Being willing to say 'I would turn it off under high concurrency' is what marks the answer as measured rather than recited.