TensorRT-LLM

NVIDIA's TensorRT-LLM compiles a model into an optimized engine for a specific GPU architecture and precision. You pay an ahead-of-time build cost. You gain tightly tuned kernels (including FP8/FP4 paths on Hopper and Blackwell), in-flight batching, and deep integration with Triton Inference Server for process management, metrics and multi-model ensembles.

It shines when the model set is relatively stable (you can afford rebuilds on upgrade), you want maximum tokens/s on NVIDIA hardware at a fixed precision, or your org already runs Triton and wants one ops surface for LLM and non-LLM models.

The big idea

A race team builds a car for one track. The car is faster than a daily driver. Changing the track means a rebuild. vLLM and SGLang are the daily drivers: eager or lightly compiled paths win on engineering time when checkpoints swap daily, even if peak FLOPs leave a few percent on the table.

The picture breaks if you do not have NVIDIA staff or a Triton habit. Then the build pipeline is cost without the ops payoff.

Walking through the trade

It is heavier to iterate on: changing max length, parallelism or precision often means rebuilding. For research fleets that swap checkpoints daily, vLLM or SGLang win on time-to-change.

Evaluate TensorRT-LLM when you are NVIDIA-locked, the model set is stable, and a few percent of tokens/s pays for the build pipeline. Pin versions. Engine upgrades change scheduler heuristics; promote with the same benchmarks you use for model changes.

Other runtimes still matter at the edges. Text Generation Inference (TGI) from Hugging Face popularized production serving for HF models; treat it as maintenance-mode relative to vLLM and SGLang for new greenfield work unless you are already deep in the HF Inference stack.

llama.cpp, ggml and GGUF target CPU, Apple Silicon and smaller GPUs. Wrong tool for a 70B multi-tenant SaaS on H100s; right tool for laptops, edge boxes and cheap CPU fallovers. See on-device inference.

MLC, Candle, ONNX Runtime GenAI and similar fill niche ports (WebGPU, mobile, Windows). Useful when the deployment target is not "Linux + CUDA cluster."

Hosted inference providers (Together, Fireworks, Groq, cloud MaaS, and the frontier labs' APIs) are engines you do not operate. Use them when utilization is spiky, ops headcount is the scarce resource, or the model is closed. Self-host when you need data residency, custom fine-tunes at high volume, or the cost-per-token math beats the API after GPU and people costs.

Common mistakes

• Rebuilding on every tokenizer tweak without pinning a digest. You cannot tell whether a regression is kernels or weights.
• Picking TRT-LLM to "go faster" on a fleet that changes max length weekly. Rebuild tax dominates.
• Using llama.cpp on H100 SaaS because it was easy on a laptop. The hardware target is the decision.

Check yourself

1. When does the compile step pay for itself? Answer: stable model set, NVIDIA hardware, and a few percent of tokens/s that matters at fleet scale.
2. What usually forces a TensorRT-LLM rebuild? Answer: changing max length, parallelism or precision.

Key takeaways

• TRT-LLM is a compiled, GPU-specific engine plus Triton ops.
• You trade iteration speed for peak NVIDIA tokens/s.
• TGI, llama.cpp and hosted APIs fill other niches, not the same job.
• Pin versions and promote with the same concurrency sweep you use for models.

Further learning

TensorRT-LLMTensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. Tensor...GitHub

Next, set the engine knobs that actually move SLOs.