Serving engines: vLLM, SGLang and TensorRT-LLM

A serving engine is the process (or process tree) that turns an HTTP request into a stream of tokens. It owns the scheduler, the KV cache, the model weights on the GPU, sampling, and the OpenAI-compatible API your product already speaks. Pick the wrong engine or leave defaults untouched and you leave 2–5× throughput on the table. Pick well and the same GPUs clear your SLO at a fraction of the fleet cost.

Every serious engine contains the same pipeline: a thin API, an iteration-level scheduler, a paged KV manager, a model executor, a sampler and a detokenizer. One forward pass scores every sequence currently scheduled. The scheduler packs those sequences so the GPU stays busy without blowing the KV budget.

Start with one engine (usually vLLM) on a single node. Nail quantization, max length and a concurrency sweep. A/B SGLang if traces show heavy prefix overlap or structured decoding is on the critical path. Evaluate TensorRT-LLM when you are NVIDIA-locked, the model set is stable, and a few percent of tokens/s pays for the build pipeline. Pin versions. Engine upgrades change scheduler heuristics.

Compare engines with a full concurrency sweep on a real length mix. Methodology is in benchmarking.

Concepts in this section

In design reviews, draw the pipeline and point to where TTFT (queue + prefill) and TPOT (decode batch size, KV bandwidth) come from. Continuous batching versus static batching, and what PagedAttention buys you, both live in this stack.

Hosted inference providers are engines you do not operate. Use them when utilization is spiky or the model is closed. Self-host when you need residency, custom fine-tunes at high volume, or the cost-per-token math wins after people costs.

Common mistakes

• Comparing engines at one concurrency and a synthetic 512/128 prompt.
• Chasing peak tokens/s instead of goodput under the product SLO.
• Turning on prefix cache without routing affinity, so every replica stays cold.
• Retrying JSON in application code instead of grammar-masked decoding in the engine.
• A parser that does not match the chat template, then blaming the model.

Key takeaways

• A serving engine is API + scheduler + KV manager + executor + sampler. Optimize scheduler and KV before buying GPUs.
• vLLM is the broad default. SGLang leads on prefix reuse and structured workloads. TensorRT-LLM leads on compiled NVIDIA peak with heavier ops.
• The knobs that matter first: tensor parallel size, max model length, GPU memory utilization, quantization, speculative decoding, prefix caching.
• Structured output should be grammar-masked in-engine, not retry loops in application code.
• Engine choice is workload-dependent. Promote changes with concurrency sweeps under a real length distribution.

Further learning

vLLM documentationdocs.vllm.ai SGLang documentationHigh-performance serving framework for large language and multimodal models.SGLang Documentation TensorRT-LLMTensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. Tensor...GitHub Inside vLLM: Anatomy of a High-Throughput LLM Inference SystemHow vLLM's inference engine works, covering PagedAttention, continuous batching, prefix caching, speculative decoding, multi-GPU serving, scheduling, and benchmvllm.ai