Speculation versus batching

Speculative decoding helps when the GPU has spare compute and users are waiting on TPOT. It hurts when the GPU is already packed. Verify forwards are wider than single-token decode: you score k positions. That raises compute per scheduling step.

Under continuous batching, those fatter steps can increase iteration time for everyone in the batch. Engines mitigate with careful batching of speculative sequences, but the qualitative point stands: speculation is not free capacity.

When it helps vs when it hurts

Helps when:

• Low batch size, latency-sensitive interactive decode (memory-bound, spare compute for verify).
• Draft acceptance is high (same domain, aligned tokenizers, good draft).
• You care about tokens per second per user, not only aggregate fleet throughput.

Hurts when:

• High batch already saturates compute — verify forwards compete with useful batch work. Speculation can reduce goodput.
• Acceptance is poor (mismatched draft, highly entropic tasks).
• Draft runs on the same scarce GPU and steals more than it returns.

Some deployments enable it only below a concurrency threshold, or only on premium queues. A 2× tok/s demo at concurrency 1 often becomes 1.1× (or 0.9×) at concurrency 32. Capstone-level systems sometimes use speculation only on the interactive priority queue and leave bulk traffic vanilla.

Prefill is usually left alone — speculation targets the decode loop. Chunked prefill and speculation compose: chunking protects TPOT from long prompts; speculation speeds the decode tokens that follow.

The three compose with quantization and batching, but they fight for the same compute budget at high concurrency. Quantization changes bytes per weight. Batching shares one weight read across users. Speculation shares one weight read across future time steps of one user.

The big idea

A bike messenger is faster than a direct truck for one letter. Put 200 letters on the truck and the messenger is in the way. Speculation is the messenger. Continuous batching is the truck. The picture breaks on a premium lane: you may still want the messenger for those few letters even while the truck is full of bulk mail.

Common mistakes

• Enabling speculation fleet-wide after a concurrency-1 win, then watching goodput fall at concurrency 32.
• Treating speculation as a throughput lever when the GPU is already compute-saturated.
• Leaving it on for JSON routes where α is poor, so every batch step gets fatter for no accepts.

Check yourself

1. The GPU is packed, batch is high, $/token is the goal. Speculate or batch harder? Answer: batch, quantize and prefix-cache first; speculation can reduce goodput here.
2. Why can a speculative step slow other users in the batch? Answer: verify scores k positions, so the shared iteration takes longer for everyone.

Key takeaways

• Speculation is a latency tool for underfilled, memory-bound decode.
• Fat verify steps can raise iteration time for the whole continuous batch.
• Enable below a concurrency threshold or only on premium queues if needed.
• Compose with chunked prefill. Measure at production concurrency.

Further learning

A Hitchhiker's Guide to Speculative Decoding (PyTorch blog)Speculative decoding is an optimization technique for inference that makes educated guesses about future tokens while generating the current token, all within a single forward pass. It incorporates a verification...PyTorch

Short visual of draft-and-verify:

Back to the speculative decoding map.