Benchmarking inference

Inference benchmarks lie easily. A blog claims "2× faster" on concurrency 1 with a warm cache and 128-token prompts, while your production p99 is a 6k-token RAG prefill under bursty arrivals. Benchmarking is how you stop arguing about anecdotes and start picking serving engines, quantization and parallelism with numbers that survive contact with traffic.

The goal is not a single tokens-per-second trophy. It is a latency–throughput curve under a workload that looks like yours, with an SLO line that marks the maximum goodput you can sell. Closed-loop sweeps find that knee quickly. Open-loop checks what happens when arrivals exceed it.

Concepts in this section

Read them in order. Each one builds on the one before.

Tools, logs and reports

Common load generators: vLLM bench serve for that engine; NVIDIA GenAI-Perf / AIPerf for Triton and TensorRT-LLM; LLMPerf for OpenAI-compatible endpoints; or a custom Locust/k6 client that speaks SSE and records token timestamps.

For each completed request, store at least: start timestamp, TTFT, mean and p95 inter-token latency, input tokens, output tokens, finish reason, model or adapter id, cache hit bytes or tokens if exposed, and error class. When TTFT is bad, split gateway queue time from engine prefill time. They have different fixes (routing vs chunked prefill / more KV).

A reproducible report has seven lines: hardware; software (engine, model digest, quant, parallel sizes, cache on/off); workload (length CDFs, sampling); method (closed vs open, ladder, warmup, duration); results (table, plot with SLO, max goodput); quality eval IDs; notes (hit rate, OOMs). Store the command lines and workload file in the PR.

On-call and capacity conversations should reference goodput under SLO, not peak tok/s. When a model upgrade is proposed, demand the same sweep on the same workload file. Saying "sweep concurrency, plot TTFT p95 against throughput, pick the knee under the SLO" is the discriminating answer.

Common mistakes

• A single tokens/s number with no SLO, no lengths and no hit rate. Treat it as marketing.
• Skipping warmup and short steps. Tails have not settled; compile is still in the sample.
• Promoting a speed win with no quality eval. Quantization and speculative decoding hide regressions.

Key takeaways

• Specify input/output length distributions and prefix structure before generating load.
• Closed-loop sweeps find the curve. Open-loop validates overload and autoscaling.
• Optimize for max goodput under SLO, not peak tokens/s.
• Warm up, measure percentiles, count real tokens, and watch for client bottlenecks and cache inflation.

Further learning

LLMPerfLLMPerf is a library for validating and benchmarking LLMs - ray-project/llmperfGitHub LLM Inference Benchmarking: Fundamental Concepts (NVIDIA)This is the first post in the large language model latency-throughput benchmarking series, which aims to instruct developers on common metrics used for LLM benchmarking, fundamental concepts…NVIDIA Technical Blog vLLM benchmarking documentationdocs.vllm.ai