How does speculative decoding get several tokens out of one target-model forward pass?
answer
- guess cheaply, check once
- one pass, many scored positions
- longest agreed prefix survives
- rejection produces a corrected token
- never zero tokens per pass
basics
~20 sA cheap drafter guesses the next few tokens. The expensive target model then scores all of those guessed positions in a single forward pass, keeps the longest prefix it agrees with, and emits one corrected token where it disagrees.
solid answer
~50 sNormal decoding emits one token per target-model forward pass. Speculative decoding inserts a cheap drafter — a small model, an n-gram lookup, or extra prediction heads — that proposes k candidate tokens. Those k tokens are appended to the context and fed to the target model as **inputs** in one pass. Because attention is causal, the logits the target produces at each drafted position are exactly what it would have produced had it generated that prefix itself, so one pass yields k+1 distributions: one per drafted position plus one for the position after the last one. Verification then walks left to right, accepting drafted tokens while the target agrees, and on the first disagreement it emits a corrected token from the target's own distribution and discards the remaining drafts. Each pass therefore emits between 1 and k+1 tokens, and the target's KV-cache entries for rejected positions are rolled back.
code
python · 17 linesdef verify(drafted, target_greedy_next, prefix):
"""Greedy verification: keep the longest prefix the target agrees with."""
out = []
for i, tok in enumerate(drafted):
want = target_greedy_next(prefix + drafted[:i])
if want != tok:
out.append(want) # corrected token; later drafts dropped
return out
out.append(tok)
out.append(target_greedy_next(prefix + drafted)) # bonus token
return out
model = {("a",): "b", ("a", "b"): "c", ("a", "b", "c"): "d"}
target = lambda ctx: model.get(tuple(ctx), "z")
print(verify(["b", "x"], target, ["a"])) # ['b', 'c'] - second draft rejectedgo deeper
Be able to say that a small fast model guesses several tokens and a big model checks them all at once, keeping the ones it agrees with. Knowing the draft never reaches the user unverified is enough at this level.
Explain the mechanics: k drafts appended as inputs, one causal forward pass producing k+1 distributions, left-to-right acceptance, a corrected token at the first mismatch, and a bonus token when everything is accepted.
Show that you know the trade being made — more FLOPs in exchange for fewer sequential steps — and mention the operational details that bite: KV rollback after rejection, the draft model competing for the same VRAM, and the serial cost of drafting itself.
Frame it as buying per-request latency out of spare arithmetic capacity, and be explicit that it does not raise a saturated fleet's throughput ceiling. Own the decision of which pools get it and what you measure to prove it earned its complexity.
## The bottleneck it attacks Autoregressive generation is inherently sequential: token t+1 cannot be computed until token t exists. So the wall-clock cost of a response is roughly *number of output tokens x cost of one model step*. At small batch sizes a decode step spends most of its time streaming weights out of GPU memory rather than doing arithmetic, which means a step that computes one token position and a step that computes six cost almost the same. Speculative decoding is the trick that converts that spare arithmetic capacity into fewer sequential steps. ## The loop, step by step 1. **Draft.** A cheap proposer generates k candidate continuation tokens. The proposer can be a small model from the same family, a lookup over tokens already in the context, or extra heads bolted onto the target model itself. "Cheap" is relative to the target: the whole scheme only works if drafting k tokens costs far less than k target steps. 2. **Verify.** The k drafted tokens are appended to the current context and run through the target model as a single forward pass. This is shaped like a small prefill, not like k decode steps. 3. **Accept or correct.** Verification compares, position by position from the left, what the target would have produced against what the drafter proposed. It accepts while they agree (under greedy decoding, "agree" means the argmax matches; under sampling, an acceptance test based on the two probabilities is used). At the first disagreement it emits a corrected token drawn from the target's own distribution at that position and throws away every remaining draft. 4. **Bonus token.** If all k drafts are accepted, the pass has also produced a distribution for the position *after* the last draft, so a free (k+1)-th token is emitted. 5. **Roll back state.** The target computed KV entries for drafted positions that were rejected. Those entries are invalid continuations and must be truncated from the cache before the next iteration. ## Why one pass is enough The key is teacher forcing plus the causal attention mask. Once the drafted tokens are known, they can be supplied as inputs, and the causal mask guarantees that position i attends only to positions <= i. The logits at position i are therefore conditioned exactly on the prefix ending at i-1 — precisely the distribution the target would have produced if it had generated that prefix step by step. Nothing about verification is an approximation of the target's behaviour; it is the target's behaviour, evaluated in parallel. ## What a pass emits Between 1 and k+1 tokens. It is never zero: even a completely wrong draft yields the corrected token at the first rejected position, so a failed speculation costs extra compute but never a wasted iteration. That floor matters operationally — a bad draft degrades the speedup toward 1x rather than stalling generation. ## What it costs - **More FLOPs, not fewer.** Every rejected drafted token was computed by the target and discarded. Speculation trades arithmetic (abundant during small-batch decode) for sequential steps (scarce). - **Serial draft steps.** Producing k drafted tokens from a draft *model* is itself autoregressive: k small forward passes that sit on the critical path. This is why very small drafters, or drafters that emit several tokens at once, are preferred. - **Memory.** A draft model brings its own weights and its own KV cache onto the same GPU, shrinking the pool available for request KV. - **Complexity.** Cache rollback, per-sequence draft state, and a second artifact to version and evaluate. ## What it does not do It does not change what the model says — with the standard verification rule the output is distributed identically to plain decoding from the target. It does not reduce total GPU work, so it does not raise the throughput ceiling of an already-busy server. And the draft model's own text is never returned to the user; the drafter only proposes candidates that the target either ratifies or overrules. ## The one number that governs everything How many tokens a pass emits on average depends on the acceptance rate — the share of drafted tokens the target keeps. High acceptance with a cheap drafter is a large latency win; low acceptance means you are paying draft cost and wasted verification FLOPs to emit roughly one token per pass, exactly as before.
- If the first drafted token is rejected, has the iteration produced nothing?No. The target's forward pass already produced a distribution at that position, so the engine samples the corrected token there and emits it. The iteration degrades to the throughput of ordinary decoding plus the wasted draft and verification work — never to zero progress. That floor is why a poor drafter slows you down gradually rather than stalling generation.
- What has to happen to the target model's KV cache after a partial rejection?The verification pass wrote KV entries for every drafted position, including the ones that were rejected. Those entries are conditioned on a continuation that will not happen, so the engine truncates the cache back to the last accepted position before the next iteration. Getting this rollback wrong silently corrupts the context, which is why speculation is an engine feature rather than something you bolt on client-side.
- Does speculative decoding raise the maximum tokens per second a saturated GPU can produce?No — it lowers latency, not the throughput ceiling. It increases total FLOPs per emitted token because rejected drafts are computed and discarded. On a GPU that already has enough concurrent work to fill its matrix units, that extra arithmetic competes with real requests, so aggregate tokens per second typically falls.
saying these in an interview costs you the question
- Says the draft model's text is returned to the user
- Claims speculation reduces total GPU compute
- Thinks verification runs the target once per drafted token
- Believes a fully rejected draft emits zero tokens
- Says it raises the throughput ceiling of a busy server