Draft and verify

Autoregressive decode emits one token per forward pass of the big model. Each pass streams the full weights from HBM for a tiny amount of math — classic memory-bound behavior from prefill vs decode. Speculative decoding asks: if HBM traffic dominates, can we get more tokens per weight read without changing what the model would have sampled?

Yes. A cheap draft mechanism proposes k tokens. The target model scores them in one parallel forward. A rejection sampling rule accepts a prefix of the draft and corrects the first rejection so the output distribution matches the target exactly.

The big idea

Verifying k candidate tokens with the target is roughly as expensive as generating one token, because both are dominated by reading weights (and attending over KV), not by the extra FLOPs along a short speculative length. Prefill-style parallelism along the speculative sequence turns those k checks into one shot.

Compare to quantization, which changes bytes per weight, and batching, which shares one weight read across users. Speculation shares one weight read across future time steps of one user.

A proofreader who checks five guessed words in one glance is faster than one who fetches the dictionary for each word. The picture breaks when the guesses are bad: you still pay the glance and keep only the first correction.

Walking through one round

Naive decode:     1 target forward → 1 token
Speculative:      1 draft path + 1 target forward → E[accepted] tokens

1. Draft: generate k tokens d1…dk from a faster source, given the current context.
2. Verify: run the target once on the context plus draft, producing probabilities at each drafted position (and one extra bonus position after the last accepted token).
3. Accept/reject left to right. Accept while the rule says yes. On first rejection, sample a corrected token from an adjusted distribution. Discard the rest of the draft.
4. Append accepted (plus corrected) tokens and repeat.

For greedy decoding the picture is simpler: accept while draft equals argmax of the target; stop at the first mismatch and take the target's argmax. Stochastic decoding needs the full rejection-sampling math. Temperature, top-p and top-k must match how the target would have sampled alone. Engines that "accept if the draft token is in the top-p set" without the proper correction are approximating, not guaranteeing identity.

Same-tokenizer alignment is non-negotiable for separate draft models. A draft that retokenizes slightly differently will tank acceptance even if the text looks identical to a human. Speculation does not shrink KV. Accepting four tokens appends four KV positions in one round.

Common mistakes

• Accepting any top-p token and calling it exact sampling. That is a heuristic.
• Mismatched tokenizers between draft and target, then blaming the algorithm.
• Expecting KV to shrink. Speculation only changes how fast you fill it.

Check yourself

1. After a rejection at draft token 4, what happens to token 5? Answer: it is discarded; the target samples a correction at position 4.
2. Does correct speculation change the target's output distribution? Answer: no, within numerics; if quality shifts, the implementation is wrong.

Key takeaways

• Draft k tokens, verify them in one target forward, accept a prefix.
• Rejection sampling keeps the target's distribution.
• Greedy is "match argmax." Stochastic needs the real correction rule.
• Tokenizer mismatch kills acceptance. KV still grows.

Further learning

Next, size the win with acceptance rate.