Benchmark pitfalls

Inference benchmarks lie easily. The usual failure is not a bad engine. It is a test that measures the wrong thing: a warm cache, a short output, a saturated client, or a slide with no SLO. This page is the checklist that keeps a number honest.

Walking through the traps

Ignoring output length. Decode is memory-bandwidth bound. Longer outputs keep batch slots occupied. A test that stops at 32 tokens will report fantasy concurrency.

Prefix cache inflation. Replaying the identical prompt 10k times with caching on measures cache hits, not prefills. Randomize suffixes, disable cache for a cold-prefill study, or report hit rate as a first-class metric.

Wrong token accounting. Billing and capacity use tokenizer tokens. Character heuristics drift 10–30% by language. Multimodal "tokens" from image tiling need the engine's accounting.

Client bottlenecks. If the load generator saturates CPU parsing SSE, you are benchmarking the client. Watch client CPU and raise worker processes until server-side KV and GPU metrics move first. The client must record TTFT from request send to the first streamed token (not the first HTTP header), record inter-token latency, and count tokenizer tokens.

Comparing different hardware silently. "Engine A on H100 vs Engine B on A100" is a hardware study, not an engine study.

One seed, one prompt. Variance across prompt shapes is huge. Use hundreds to thousands of prompts drawn from the target distribution.

Measuring only mean latency. Means hide the queue blow-ups that page your on-call. Always publish p95/p99 for TTFT and TPOT.

Comparing engines fairly

When A/B testing vLLM vs SGLang vs TensorRT-LLM:

• Same GPUs, driver, model weights digest, tokenizer, max length and parallel size.
• Same workload file and sampling parameters.
• Same SLO definition and goodput rule.
• Enable each engine's best supported equivalent features (prefix cache, FP8), and document parity gaps. An engine without prefix cache is not "slower" on unique prompts; it is slower on shared prompts.
• Run at least two non-overlapping windows or two nodes to catch thermal and clock variance.

Gate every speed win with a quality eval, especially for quantization and speculative decoding. A config that is 15% faster and fails 3% more tool calls is usually a net product loss. Spot-check speculative acceptance rate; a drop often predicts both quality drift and wasted draft FLOPs.

Common tools: vLLM bench serve for that engine; NVIDIA GenAI-Perf / AIPerf for Triton and TensorRT-LLM; LLMPerf for OpenAI-compatible endpoints; or a custom Locust/k6 client that speaks SSE correctly.

Common mistakes

• Trusting only the load generator's summary slide. Store per-request TTFT, ITL, tokens, finish reason and error class, then aggregate offline.
• Leaving prefix cache on by accident. Report hit rate or turn it off for a cold-prefill study.
• Promoting a speed win with no eval suite. Quantization and speculative decoding hide quality regressions.

Check yourself

1. You replay one prompt 10k times with prefix cache on. What did you measure? Answer: cache hits, not prefills.
2. Engine A on H100 beats engine B on A100. Can you conclude A is faster? Answer: no. That is a hardware study unless the GPUs match.

Key takeaways

• Count tokenizer tokens, measure percentiles, and watch the client.
• Report cache hit rate, length mix, concurrency and the SLO next to every tok/s number.
• Compare engines on the same hardware, workload and goodput rule, then gate speed with quality.

Further learning

LLMPerf and the vLLM benchmarking docs are the usual starting tools. Back to the benchmarking map.