SGLang

SGLang (LMSYS) optimizes for prefix-heavy and structured workloads. Its signature is RadixAttention: a radix tree over token prefixes so the KV cache is shared across requests the way a filesystem shares inode prefixes. Cache-aware scheduling tries to keep high-hit prefixes warm and route work so reuse stays high.

For agent loops, tool calling, and apps that resend the same long system prompt, this is often the difference between compute-bound prefills and near-zero TTFT on cache hits.

Walking through the product fit

SGLang also invests in a low-overhead scheduler and fast constrained decoding. If your product needs JSON schemas, grammar-constrained tool calls, or high QPS of short completions against shared templates, measure SGLang alongside vLLM on your trace — not a generic ShareGPT dump.

There is a frontend language for composing multi-call programs (generate, choose, fork). That matters when one user action fans out into many model calls that share context. Even if you never write that DSL, the runtime underneath is built for that reuse pattern.

Constrained decoding is not free. Naive "sample then validate JSON and retry" burns tokens and latency. Engines that compile a grammar or JSON schema into a token mask (XGrammar and related backends in vLLM and SGLang) keep the batch GPU-efficient while guaranteeing structure.

A tool-calling agent that emits {"name":"...","arguments":{...}} every turn is the usual example. Unconstrained sampling with a retry loop might average 1.3 attempts and add 80–200 ms of TTFT variance. XGrammar-style masking accepts the schema on the first pass. The remaining cost is a modest per-step mask overhead, usually cheaper than a second full generation. Benchmark tool-call p95 separately from free-form chat — they are different workloads.

Tool-call parsers on the API side must stay in sync with the model's chat template. A wrong parser silently drops arguments. That bug looks like a model failure and wastes days.

The big idea

Think of a filing cabinet labeled by the start of the filename. Every file that begins with the same system prompt shares a drawer. A one-off user turn lives in a thin folder at the end. The picture breaks when routing sprays those files across cabinets: the tree on each replica stays cold. Pair SGLang with cache-aware routing.

Common mistakes

• Benchmarking on unique ShareGPT prompts and concluding RadixAttention does nothing.
• Retrying JSON in application code instead of grammar-masked decoding.
• Shipping a chat-template parser that does not match the model, then blaming the weights.

Check yourself

1. When does SGLang usually beat a generic vLLM setup on cost? Answer: when many requests share long prefixes, so RadixAttention turns prefills into cache hits.
2. Why is "sample then validate JSON" expensive? Answer: failed attempts burn a full generation; a token mask accepts structure on the first pass.

Key takeaways

• RadixAttention shares KV along token prefixes.
• SGLang is the usual pick when prefix hit rate or structured output is the product constraint.
• Grammar-masked decoding beats retry loops.
• Measure on your trace, including tool-call p95.

Further learning

SGLang documentationHigh-performance serving framework for large language and multimodal models.SGLang Documentation

Next, see when a compiled NVIDIA engine is worth it in TensorRT-LLM.