Speculative decoding: making LLM inference faster without changing the output

A transformer generates text one token at a time: to produce token t it must have already produced t-1. Each token is a full forward pass over billions of parameters, and at batch size 1 that pass is memory-bandwidth bound, the GPU spends most of its time streaming weights from memory, not doing math. So the arithmetic units sit mostly idle, and end-to-end latency scales almost linearly with the number of tokens.

The key observation: verifying a proposed sequence is cheap in a way that generating it is not. If you already had a candidate continuation, one forward pass could score every position in it at once, using the idle compute the sequential loop wastes. Speculative decoding is built entirely on turning generation into verification.

Speculative decoding runs two models. A small, fast draft model q guesses the next few tokens cheaply, autoregressively. Then the large target model p, the one whose output you actually want, checks all of those guesses in one parallel forward pass.

Because the target scores $\gamma$ drafted positions plus one more in a single pass, a good run yields several tokens for the price of one target step. A bad run (the draft guessed wrong early) falls back to roughly one token, the same as ordinary decoding. So the technique never makes you slower in expectation; it only ever adds tokens-per-pass when the draft is right.

The whole game is: make the draft cheap, make it agree with the target often, and verify in a way that doesn't change what the target would have produced on its own.

The draft proposes; the target verifies in parallel; a rejection corrects losslessly.

The magic is the acceptance rule. Walk the draft left to right. For a drafted token x, accept it with probability

$\alpha = \min\!\left(1, \frac{p(x)}{q(x)}\right)$

where $p(x)$ and $q(x)$ are the target and draft probabilities for that token in that context. If the target likes the token at least as much as the draft did ($p \ge q$), accept outright. If it likes it less, accept only proportionally.

On the first rejection, discard the rest of the draft and resample that one position from the corrected residual distribution, the normalized positive part of $p - q$:

$x \sim \operatorname{norm}\big(\max(0,\; p - q)\big)$

Stern, Leviathan, and Chen proved this makes the whole scheme lossless: the tokens you emit are distributed exactly as if you had sampled from the target p alone. Greedy decoding matches token-for-token; sampling matches in distribution.

One target forward pass now yields more than one token whenever the draft's leading guesses survive verification. If each drafted token is accepted independently with rate $\alpha$ and you draft $\gamma$ tokens per step, the expected number of tokens emitted per target pass is

$\mathbb{E}[\text{tokens/pass}] = \frac{1 - \alpha^{\gamma+1}}{1 - \alpha}$

At $\alpha = 0.8$ and $\gamma = 4$, that is about 3.4 tokens per target pass instead of 1, a large latency win.

Crucially, the win comes from the memory-bandwidth slack of the batch-1 regime. The extra drafted positions ride along in the same weight-streaming pass, so verifying 5 positions costs barely more than verifying 1. The gains are real but bounded by two things: how often the draft agrees with the target ($\alpha$), and how cheap the draft is relative to the target.

The loop makes the accept/reject rule concrete. Note the single parallel target_p(...) call over all drafted positions, and the break on first rejection:

# one speculative step: draft with q, verify with target p
draft = []
for _ in range(gamma):                 # draft gamma tokens (cheap, sequential)
    draft.append(sample(q(context + draft)))

p = target_p(context + draft)          # ONE parallel pass -> [gamma+1, vocab]

out = []
for j, tok in enumerate(draft):
    if uniform(0, 1) < min(1, p[j][tok] / q[j][tok]):
        out.append(tok)                # accept
    else:                              # reject: resample the residual, then stop
        out.append(sample(normalize(relu(p[j] - q[j]))))
        break
else:
    out.append(sample(p[gamma]))       # all accepted -> free bonus token

That for/else is doing real work: the else runs only when the loop finished with no break, i.e. every drafted token was accepted, so you also collect the target's bonus token.

How many tokens should the draft propose per step? The parameter $\gamma$ is a tradeoff, not a free dial.

• Larger $\gamma$ raises the ceiling on tokens-per-pass, but acceptance is compounding, so the marginal token is accepted only with probability $\alpha^k$, which decays fast. Beyond a point you are mostly paying draft cost for tokens that get rejected.
• Smaller $\gamma$ wastes less draft compute per rejection but caps your best-case speedup.

The sweet spot depends on the acceptance rate: a well-aligned draft ($\alpha$ near 0.9) rewards a longer $\gamma$; a weak draft wants a short one. Many systems make $\gamma$ adaptive, shortening it when recent acceptance drops. The draft is pure overhead on any step where the first token is rejected, so a draft that is both cheap and frequently right matters more than a long one.

The original recipe (Leviathan et al.; Chen et al.) uses a separate small language model as $q$, say a 1B model drafting for a 70B target. It is simple and modular: swap the draft freely.

Two practical constraints:

• Shared vocabulary. Acceptance compares $p(x)$ and $q(x)$ token-for-token, so the draft and target must tokenize identically. A draft with a different tokenizer needs remapping and loses alignment.
• Alignment beats raw quality. What matters is that $q$ agrees with $p$ where $p$ is confident, not that $q$ is good in absolute terms. Distilling the draft from the target's own outputs raises the acceptance rate more than simply using a stronger off-the-shelf small model.

The cost is operational: you now serve and keep two models resident in memory.

Serving a second model is annoying, so a family of methods draws the draft from the target, no separate network to deploy:

Each trades a bit of acceptance rate for operational simplicity. Medusa and self-speculation keep everything inside one served model; lookup drafting adds essentially no compute at all and shines when the output echoes the input (code edits, summarization with quotes).

A linear draft is one bet: a single chain of $\gamma$ tokens. If token 2 is wrong, tokens 3 onward are wasted. Tree drafting hedges by proposing several candidate continuations arranged as a tree, then verifying all branches in one pass using a masked tree attention pattern (each node attends only to its ancestors). More of the tree survives, so more tokens are accepted per step.

The EAGLE line pushes this hard. Instead of predicting tokens, it autoregresses at the feature level and conditions the draft on the target's own deeper hidden states, which makes its guesses agree with the target far more often. Combined with draft trees, EAGLE-2 and EAGLE-3 report acceptance rates high enough for roughly 3-4x lossless speedups. The pattern generalizes: richer, better-aligned draft structures buy more accepted tokens per verification, at the cost of a bit more verify-time compute.

Speculative decoding is one of the few free lunches in inference, but the lunch has a shape.

• Lossless, genuinely. With correct rejection sampling the output distribution is identical to the target's. This is not quantization or distillation; quality does not move.
• A latency win, not always a throughput win. The speedup comes from spare compute in the memory-bandwidth-bound, low-batch regime. At high batch size the target pass is already compute-bound, so the extra drafted positions compete for the same busy units and the gain shrinks, sometimes to nothing.
• It depends on agreement. Off-distribution inputs, high sampling temperature, or a poorly aligned draft drop $\alpha$, and with it the benefit.
• It adds moving parts. A draft model or extra heads, more memory, and verification logic in the serving stack.

So it is a latency accelerator for interactive, low-batch serving, not a universal throughput multiplier.

Strip away the specific methods and one idea remains: turn expensive sequential generation into cheap parallel verification, and correct the guesses in a way that changes nothing. Everything else, a separate draft model, Medusa heads, layer-skipping, EAGLE's feature trees, is just a different way to produce guesses that the target will accept often.

That framing is why speculative decoding became standard plumbing: modern serving engines (vLLM, TensorRT-LLM, SGLang, TGI) ship it, and new draft methods slot into the same accept-and-correct backbone without touching the correctness proof. When you evaluate a new variant, ask the two questions the math forces: how cheap is the draft and how often does the target accept it. Those two numbers, not the branding, determine the speedup.