Serving systems
Engine knobs that matter
Tensor parallel size, max model length, GPU memory utilization, quantization, speculation and prefix caching. Ignore the long flag list until these are right.
Ignore the long flag list until these six knobs are right. They control whether the GPU holds enough KV, whether collectives eat your decode, and whether repeated prompts prefill again.
| Knob | What it controls | Failure mode if wrong |
|---|---|---|
tensor_parallel_size | Shards weights across GPUs | Too high → communication tax; too low → OOM |
max_model_len | Max context; sizes KV pool | Too high → fewer concurrent seqs; too low → 400s in prod |
gpu_memory_utilization | Fraction of VRAM for engine | Too high → OOM under fragmentation; too low → idle HBM |
| Quantization | Weight and KV precision | Too aggressive → quality regressions users notice |
| Speculative decoding | Extra draft compute | Helps TPOT when acceptance is high; hurts when low |
| Prefix caching | KV reuse across requests | Off → repeated prefills; on without routing → uneven hits |
Walking through the defaults
Tensor parallel size. For a dense 70B in FP8, 2-way TP on a single node is common; 8-way for larger models or longer contexts. Pipeline parallel appears when the model does not fit in one node's memory even sharded. Expert parallel is for MoE — see MoE serving.
Max model length. Product owners ask for "128k context" by default. Every unused kilotoken of max length reserves KV headroom that could have been concurrency. Set max to the p99 you actually need, and put long-tail jobs on a separate pool.
GPU memory utilization. Start around 0.85–0.90, load a realistic concurrency sweep, watch for OOMs during long-context spikes, then nudge. Leave margin for CUDA graphs, fragmentation and the occasional multimodal activation spike.
Quantization. Weight-only INT4/AWQ/GPTQ and FP8 are table stakes for 70B-class models on 80 GB cards. Quantizing the KV cache further multiplies concurrency. Always keep a quality eval gate — see quantization and evals.
Speculative decoding. Worth enabling when draft acceptance rates stay high on your traffic (code continuation often does; highly creative open-ended chat often does not). Measure TPOT and acceptance together. Speculation that rejects often is pure overhead. See speculative decoding.
Prefix caching. Turn it on for any app with shared system prompts. Then make sure routing sends similar prefixes to the same replica, or the cache hit rate collapses across a horizontally scaled fleet.
Start with one engine (usually vLLM) on a single node. Nail quantization, max length and a concurrency sweep. A/B SGLang if traces show heavy prefix overlap or structured decoding is on the critical path. Evaluate TensorRT-LLM when you are NVIDIA-locked and a few percent of tokens/s pays for the build pipeline.
Export metrics the gateway can use: queue depth, KV utilization, prefix hit rate, active LoRAs. Routing without these signals is round-robin with extra steps.
Warning
Never compare engines with a single concurrency and a fixed 512/128 token synthetic prompt. Engine A may win at concurrency 8 and lose at 64. Prefix caching may look miraculous on repeated prompts and useless on unique ones. Use the benchmarking methodology — same workload file, same SLO, full sweep.
Common mistakes
- Setting 128k max length for a product whose p99 is 4k.
- Enabling speculation globally when JSON tool calls reject most drafts.
- Turning on prefix cache but routing at random, so every replica stays cold.
Check yourself
- Why does unused max length cost throughput? Answer: it reserves KV that cannot become concurrent sequences.
- Which two signals should the gateway see besides request count? Answer: KV utilization and prefix hit rate (and queue depth).
Key takeaways
- Six knobs dominate: TP size, max length, memory utilization, quantization, speculation, prefix cache.
- Set max length from p99, not from a brochure.
- Prefix cache needs routing affinity or the hit rate dies.
- Promote engine changes with a concurrency sweep under a real length mix.
Further learning
Back to the serving engines map, then measure with benchmarking.
Related
Serving engines: vLLM, SGLang and TensorRT-LLM
How modern engines are built, how they differ, and which knobs move TTFT, TPOT and tokens per GPU-second.
Quantization
INT8, FP8, INT4 and FP4. Fewer bits shrink weights, activations and KV. Match the format to memory-bound decode or compute-bound prefill, then eval quality.
Speculative decoding
Draft-and-verify so one target forward accepts several tokens. Same output distribution. Real speedups when decode is memory-bound and acceptance stays high.