Speculative decoding

Autoregressive decode emits one token per forward pass of the big model. Each pass streams the full weights from HBM for a tiny amount of math — classic memory-bound behavior from prefill vs decode. Speculative decoding keeps the sampling semantics of the big model but changes the hardware schedule so each weight read does more useful verification work.

A cheap draft proposes k tokens. The target scores them in one parallel forward. Rejection sampling accepts a prefix and corrects the first rejection so the output distribution matches the target exactly. When drafts are often right, you multiply throughput. When they are wrong, you still make progress (at least one correct token) but pay draft overhead. The art is keeping acceptance high on your traffic without starving the rest of the batching system.

If expected accepted tokens per round exceed 1 + cost_ratio, you win. It is a systems trick layered on unchanged model weights (except when you add Medusa, EAGLE or MTP heads).

Ship speculation behind a flag and an SLO experiment. A 2× tok/s demo at concurrency 1 often becomes 1.1× (or 0.9×) at concurrency 32. Fewer GPUs for the same interactive SLO is the win, not a vanity tok/s chart.

Concepts in this section

Quantization changes bytes per weight. Batching shares one weight read across users. Speculation shares one weight read across future time steps of one user. The three compose, but they fight for the same compute budget at high concurrency.

Interview angle: explain memory-bound verify, draw draft/verify/accept, state distribution preservation, and give the α–k intuition with a worked numeric sketch.

Common mistakes

• Trusting a concurrency-1 speedup and enabling speculation fleet-wide.
• A heuristic "accept if in top-p" that silently changes the distribution.
• Mismatched tokenizers between draft and target, then blaming the algorithm.
• Leaving it on for JSON routes where acceptance is poor and every batch step gets fatter.

Key takeaways

• Speculative decoding drafts several tokens and verifies them in one target forward to amortize weight reads.
• Rejection sampling keeps outputs distributionally faithful to the target model.
• Speedup hinges on acceptance rate α, draft length k, and draft overhead. Measure on real traffic.
• Drafts can be small models, n-grams, Medusa/EAGLE heads, or native multi-token predictors.
• Best for low-batch latency. Can hurt when the GPU is already compute-saturated.
• Configure via serving-engine speculative settings and validate at production concurrency.

Further learning

Fast Inference from Transformers via Speculative DecodingInference from large autoregressive models like Transformers is slow - decoding K tokens takes K serial runs of the model. In this work we introduce speculative decoding - an algorithm to sample from autoregressive mo…arXiv.org EAGLE: Speculative Sampling Requires Rethinking Feature UncertaintyAutoregressive decoding makes the inference of Large Language Models (LLMs) time-consuming. In this paper, we reconsider speculative sampling and derive two key observations. Firstly, autoregression at the feature (se…arXiv.org Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding HeadsLarge Language Models (LLMs) employ auto-regressive decoding that requires sequential computation, with each step reliant on the previous one's output. This creates a bottleneck as each step necessitates moving the fu…arXiv.org A Hitchhiker's Guide to Speculative Decoding (PyTorch blog)Speculative decoding is an optimization technique for inference that makes educated guesses about future tokens while generating the current token, all within a single forward pass. It incorporates a verification...PyTorch

Short visual of draft-and-verify: