Speculative Decoding

Large language models write one word at a time. To produce the next token, the model runs a full forward pass over billions of parameters; then it appends that token and does it all again for the next one. A 500-token answer means 500 sequential passes through the whole network. That strict left-to-right dependency is the reason generation feels slow — and, surprisingly, it is mostly waste.

Speculative decoding is a trick that gets the exact same output out of far fewer passes through the big model. Here is the whole idea in one figure — the same sentence, produced two ways, racing on the same clock:

Both rows end at the identical text. The autoregressive row spends one forward pass per token. The speculative row produces the text in bursts, finishing in less than half the passes. The rest of this article explains how that is possible — and why it gives back exactly the same distribution, not an approximation.

Why one-token-at-a-time is wasteful

On modern accelerators, generating a single token from a large model is memory-bandwidth bound, not compute bound. The bottleneck is reading the model’s weights out of memory; once they are loaded, the matrix multiplies are cheap and the hardware is mostly idle.

Here is the key asymmetry that follows from that:

Scoring one candidate token costs almost exactly the same as scoring ten of them in a single forward pass.

Generation is sequential because each token depends on the previous one. But verification is not. If someone hands you a draft of the next ten tokens, the big model can check all ten in one pass — at essentially the cost of producing one. That gap between cheap parallel verification and expensive sequential generation is the entire opportunity.

Draft, then verify

Speculative decoding exploits the asymmetry with two models:

• A small, fast draft model $q$ (e.g. a 1B model, or a distilled version of the target) that guesses the next $k$ tokens cheaply and autoregressively.
• The large target model $p$ — the model whose output you actually want — which scores all $k$ guesses in a single forward pass.

Each round then does three things:

1. The draft model proposes $k$ tokens.
2. The target model verifies them in one pass, producing a probability for each.
3. An accept/reject rule keeps the longest correct prefix and corrects the first mistake. The output advances by (accepted tokens + 1), at the cost of one target pass.

Step through a real trace below — every probability here was computed offline by running an actual draft (distilgpt2) and target (gpt2-medium) model; the page just replays it. Watch the draft model spray out guesses, the target turn them green or red, and the token-per-pass counter climb above 1.

When the draft is right, you get several tokens for the price of one pass. When it is wrong, you fall back to one token — never worse than plain autoregressive decoding, which is what makes the technique a strict win in pass count.

Why it stays exact

The surprising part is that this is not an approximation. The accepted text is distributed identically to what you would get by sampling from the target model $p$ alone. The magic is in the accept/reject rule.

For each draft token $x$, compare how much the target wanted it, $p(x)$, to how much the draft proposed it, $q(x)$:

If the target likes the token at least as much as the draft did, accept it outright. If the target likes it less, accept it only sometimes. On the first rejection, throw away the rest of the draft and resample a single replacement token from the leftover probability mass:

This “modified rejection sampling” exactly cancels the bias introduced by drafting with the wrong distribution $q$. Explore it on a real step below — each draft token’s draft probability $q$, target probability $p$, and resulting acceptance chance:

If all $k$ tokens are accepted, the target’s pass also gave us a free next-token distribution, so we sample one bonus token from it. Either way the round emits at least one token, and the math guarantees the same output distribution as the target.

How much speedup, really?

The win depends on the acceptance rate $\alpha$ — how often the draft agrees with the target — and the lookahead $k$. If draft tokens were free, each target pass would yield, on average,

which climbs toward $k+1$ as the draft gets better. But the draft is not free: it costs some fraction $c$ of a target pass per token, and that tax sits in the denominator. Bigger $k$ helps only until the draft cost overwhelms the gains — so there is an optimal lookahead for every acceptance rate. Play with it:

The amber ring marks the real distilgpt2 → gpt2-medium run from the trace above: about a 62% acceptance rate, for a 2.45× reduction in target passes. Production systems pair a well-matched draft with a strong target and routinely see 2–3× wall-clock speedups with zero change to the output.

When it helps (and when it doesn’t)

Speculative decoding shines exactly when single-token generation is memory-bound — low batch sizes, interactive latency-sensitive serving, big models. Its leverage shrinks when:

• The batch is already large. With many sequences in flight the target’s matmuls become compute-bound, so “free” verification is no longer free.
• The draft is poorly matched. A low acceptance rate means lots of rejected work; the draft tax can erase the benefit.
• The draft is too big. A slow draft costs nearly as much as the target it is trying to save.

The same accept/verify skeleton underlies a whole family of newer methods — self-speculation, Medusa-style multi-head drafting, n-gram and lookahead decoding, EAGLE — all chasing a higher acceptance rate or a cheaper draft. But they share the one guarantee that makes the original so appealing: the output you get is the output the big model would have produced anyway — just with most of the sequential waiting removed.

Every number and probability in this article comes from a real speculative-decoding run computed offline; the page ships only the resulting traces, no model in the browser. See precompute/ for the pipeline. Foundational papers: Leviathan, Kalman & Matias (2023) and Chen et al. (2023).