Anatomy of a serving engine

A serving engine is the process (or process tree) that turns an HTTP request into a stream of tokens. It owns the scheduler, the KV cache, the model weights on the GPU, sampling, and the OpenAI-style API. Every production engine looks roughly like this pipeline, even when the code is folded into fewer modules.

Walking through the parts

API server. Accepts OpenAI-style /v1/chat/completions (and often completions, embeddings, transcription). Parses JSON, applies auth and rate limits if you put them here, streams Server-Sent Events back. Keep this layer thin. Expensive work belongs behind the scheduler.

Scheduler. Decides which requests run in the next forward pass. Modern engines use iteration-level (continuous) batching: after every decode step, finished sequences leave and waiting ones join. Prefills are usually chunked so a long prompt cannot monopolize the GPU for hundreds of milliseconds.

KV cache manager. Allocates and frees key/value blocks. With PagedAttention, memory is virtualized into fixed-size blocks so you do not reserve a contiguous max-length slab per request. Prefix hashing lets two requests that share a system prompt reuse the same blocks.

Model executor. Runs the transformer layers: attention, MLP, MoE routing, residual norms. CUDA graphs, torch.compile, FlashAttention and tensor-parallel collectives live here. On multi-GPU, the executor shards weights and synchronizes activations each layer.

Sampler. Turns logits into the next token ID: temperature, top-p, top-k, penalties, stop strings, logit bias. Structured-output engines (XGrammar, outlines-style FSMs) constrain the allowed token set each step without discarding the whole batch.

Detokenizer. Maps token IDs to UTF-8 text incrementally so the client sees partial words as soon as they form, not only at EOS.

The scheduler's job is to pack sequences into each step so the GPU stays busy without blowing the KV memory budget.

The big idea

A kitchen ticket rail decides who cooks next, the pantry tracks ingredients, and the stove runs one ticket-batch at a time. The engine is that kitchen. The analogy breaks when one "ticket" is a 32k prefill: without chunking, it occupies the stove and every short order waits.

Common mistakes

• Putting rate limits only in the API process and calling that "scheduling." Admission still has to see KV headroom.
• Reserving a max-length KV slab per request. That is the waste PagedAttention exists to stop.
• Sampling in application code after the engine. You lose batching and pay an extra round trip.

Check yourself

1. After a decode step, a request finishes and a waiter starts. What made room? Answer: the iteration-level scheduler dropped the finished sequence and admitted the waiter into the next forward.
2. Why chunk prefills instead of running a 32k prompt in one shot? Answer: an unchunked prefill monopolizes the GPU and stalls every decode in the batch.

Key takeaways

• The engine is API + scheduler + KV manager + executor + sampler + detokenizer.
• A forward pass scores every scheduled sequence at once.
• Continuous batching and paged KV are what keep the GPU busy without wasting memory.
• Keep the API thin. Policy that needs GPU state belongs in the scheduler.

Further learning

Next, see how vLLM implements this pipeline.