Speculative acceptance rate

Let α be the probability a drafted token is accepted (simplified i.i.d. model — reality is position-dependent). With draft length k, expected accepted drafted tokens before the first rejection is a geometric-like sum. You always emit at least the corrected token at the rejection point (or a bonus token if all k are accepted, depending on the variant).

E[tokens per round] ≈ (1 - α^(k+1)) / (1 - α)   # under simplified assumptions

If expected accepted tokens per round exceed 1 + cost_ratio, you win. Sweep k. Too small → little upside. Too large → wasted draft work after early rejection and a heavier verify. Typical sweet spots are small single digits unless the draft is extremely aligned.

Walking through two alphas

Treat each drafted position as succeeding with probability α, independently (a teaching lie that is still useful for sizing).

alpha, k = 0.8, 4
accepted = sum(alpha ** i for i in range(1, k + 1))
print(round(accepted, 4))

Output:

2.3616

P(accept at least 1 draft) = 0.8
P(accept at least 2)       = 0.64
P(accept at least 3)       = 0.512
P(accept all 4)            = 0.4096
Plus correction / bonus   → often ~3+ tokens delivered per target forward

If drafting those 4 tokens costs 30% of a target forward, net speedup lands near 3 / 1.3 ≈ 2.3× before real-world inefficiencies.

Suppose α = 0.7, k = 5, and a target forward costs ~1.0 units while drafting five tokens costs ~0.25 units. Rough accepted length often lands around ~3 tokens/round in this regime (measure; do not trust the napkin blindly). Cost per round ≈ 1.25 → effective cost per token ≈ 1.25 / 3 ≈ 0.42 → about 2.4× vs baseline cost 1.0 per token.

Drop α to 0.4 and E[draft accepts] for k=4 becomes 0.4+0.16+0.064+0.0256=0.65 — barely worth it once overhead is counted. If α falls to 0.3 because draft and target disagree on code tokens, k=5 can become a net loss.

The engineering job is less "implement reject sampling" (engines have that) and more "raise α on our traffic" and "pick k from a sweep" at target concurrency.

Patterns you will see in vLLM, SGLang and TensorRT-LLM (names vary by version): enable speculative mode, point at a draft model or Medusa/EAGLE/MTP heads, set num_speculative_tokens (k). Benchmark TTFT (usually little changed) and TPOT at your concurrency. Measure acceptance rate if the engine exposes it. An acceptance rate that looks great on newsy English and terrible on JSON tool calls means you need a different draft or should disable speculation on those routes.

Common mistakes

• Picking k=8 because a paper did. Early rejection wastes the tail.
• Trusting a concurrency-1 demo. 2× at batch 1 often becomes 1.1× or 0.9× at concurrency 32.
• One global α. Chat, code and JSON are different drafts.

Check yourself

1. For α = 0.8 and k = 4, what is E[draft tokens accepted]? Answer: 0.8 + 0.64 + 0.512 + 0.4096 = 2.3616.
2. When does raising k start to hurt? Answer: when extra drafted tokens are usually rejected, so draft and verify cost grow faster than accepts.

Key takeaways

• Speedup is accepted tokens per round divided by draft-plus-verify cost.
• α and k dominate. Sweep k in {2,3,4,5,8} at production concurrency.
• Measure α by route. Disable where it is poor.
• Napkins assume i.i.d. accepts. Reality is position-dependent — still useful for sizing.

Further learning

Next, pick a draft source that can actually raise α.