vLLM

vLLM popularized PagedAttention and continuous batching for open models. The V1 engine (the default path as of when this was written) is a redesign around a unified iteration-level scheduler. Each step has a token budget (max_num_batched_tokens, often on the order of 2k–8k depending on model and GPU). Running requests are scheduled first, then waiting requests are admitted.

Long prompts are chunked and mixed with single-token decodes in the same batch, so TTFT for new arrivals does not wait for a giant prefill to finish.

Walking through a typical process tree

Internally you typically see:

• A frontend API process that owns HTTP and tokenization.
• An engine core that runs the scheduler loop and talks to GPU workers.
• Workers that hold weights, run the compiled or graphed model, and manage physical KV blocks.

CUDA graphs and torch.compile cut Python and launch overhead on the decode-heavy path. Prefix caching is automatic when enabled: the engine hashes token block sequences and reuses KV for matching prefixes. Speculative decoding (n-gram, EAGLE-style draft models, and related methods) plugs into the same scheduler by asking for multiple tokens per sequence when the draft is accepted.

vLLM's strength is breadth: Hugging Face model zoo coverage, multi-LoRA, multimodal, OpenAI and related APIs, and a large community. When a new open model ships, a working vLLM recipe usually appears within days.

Prefer vLLM as the default when you need widest model support, multi-LoRA platforms, or the largest ops community. Re-benchmark when a major scheduler rewrite ships — the winner on your workload in 2024 is not guaranteed later.

The big idea

vLLM is a busy restaurant with a ticket rail that reseats tables every course, and a pantry that shelves leftovers in labeled bins. The bins are paged KV. The reseating is continuous batching. The picture breaks when your traffic is almost all shared prefixes and structured JSON: SGLang is built to win that pattern.

Common mistakes

• Leaving max_model_len at 128k "just in case." Unused context reserves KV that could have been concurrency.
• Raising gpu_memory_utilization to 0.98 and calling OOMs under long-context spikes a model bug.
• Comparing vLLM to another engine at one concurrency and a fixed 512/128 synthetic prompt.

Check yourself

1. What does max_num_batched_tokens cap? Answer: how many tokens, prefill plus decode, share one scheduler step.
2. Why can raising memory utilization beat another quantization notch? Answer: extra HBM becomes more concurrent sequences, which raises decode intensity.

Key takeaways

• vLLM's V1 path is an iteration-level scheduler with a token budget and chunked prefill.
• PagedAttention plus prefix hashing is the default memory story.
• Breadth of models and ops community is the reason it is the usual first engine.
• Memory utilization and max length move concurrency more than most flags.

Further learning

vLLM documentationdocs.vllm.ai Inside vLLM: Anatomy of a High-Throughput LLM Inference SystemHow vLLM's inference engine works, covering PagedAttention, continuous batching, prefix caching, speculative decoding, multi-GPU serving, scheduling, and benchmvllm.ai

Next, compare SGLang on prefix-heavy traces.