Draft sources

The draft can come from a second model, from the prompt itself, or from extra heads trained on the target. Pick based on stack support. vLLM, SGLang and TensorRT-LLM all ship some speculative modes. Check which draft types your version actually supports before promising latency wins.

Prefer draft checkpoints from the same family (e.g. 8B beside 70B of one line) or heads trained on the target's hidden states (EAGLE, Medusa, MTP) so the vocabulary and features match by construction. Same-tokenizer alignment is non-negotiable for separate draft models.

The big idea

A junior who trained in the same shop as the senior guesses the next sentence better than a stranger. Extra heads on the senior (Medusa, EAGLE, MTP) are even closer: they share the same brain state. The picture breaks when the "junior" uses a different tokenizer. The words look the same to you and different to the model, so acceptance collapses.

Walking through the choice

Small LM. Classic and flexible. You pay extra weights on the same GPU (or a sidecar) and a sync between draft and verify. Need a similar training distribution or α dies on domain shift.

Prompt lookup / n-gram. Almost free. Helps when the model is repeating or copying from context (code, RAG quotes). Useless on high-entropy chat.

Medusa. Extra decoding heads on the target. No second model to ship. You must train the heads, and depth is limited.

EAGLE. Drafts from target features with a lightweight head. Often high acceptance. More moving parts to implement and upgrade.

MTP. Models trained to predict several future tokens (DeepSeek-style heads). Strong alignment because the target was trained for it. Only works if the checkpoint has the heads.

Also budget KV growth. Accepting four tokens appends four KV positions in one round. Speculation does not shrink KV.

Ship speculation behind a flag and an SLO experiment. Operational checklist:

1. Baseline TPOT/goodput without speculation at production concurrency.
2. Enable a supported draft mode. Sweep k in {2,3,4,5,8}.
3. Track acceptance rate by route (chat vs code vs JSON).
4. Disable or route-select where α is poor.
5. Re-run cost models — fewer GPUs for the same interactive SLO is the win, not a vanity tok/s chart.

Common mistakes

• Promising a draft mode the engine version does not ship.
• Pairing a 7B from another family with a 70B target and a different tokenizer.
• Using n-gram drafts on open-ended chat and concluding speculation "doesn't work."

Check yourself

1. Which draft is almost free but only helps when text repeats? Answer: prompt lookup / n-gram.
2. Why do EAGLE or MTP heads often beat a random small LM on α? Answer: they share the target's states or were trained to predict its future tokens, so vocab and features match.

Key takeaways

• Drafts are small LMs, n-grams, Medusa, EAGLE or native MTP heads.
• Same family or same hidden states beat a random small model.
• Check engine support before you promise a win.
• Raise α on your traffic. That is the real work.

Further learning

Fast Inference from Transformers via Speculative DecodingInference from large autoregressive models like Transformers is slow - decoding K tokens takes K serial runs of the model. In this work we introduce speculative decoding - an algorithm to sample from autoregressive mo…arXiv.org EAGLE: Speculative Sampling Requires Rethinking Feature UncertaintyAutoregressive decoding makes the inference of Large Language Models (LLMs) time-consuming. In this paper, we reconsider speculative sampling and derive two key observations. Firstly, autoregression at the feature (se…arXiv.org Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding HeadsLarge Language Models (LLMs) employ auto-regressive decoding that requires sequential computation, with each step reliant on the previous one's output. This creates a bottleneck as each step necessitates moving the fu…arXiv.org

Next, see when speculation fights the batch.