The inference stack

Think of serving as stacked layers. A bug or a win usually lives in one of them. If a decode step is slow because Python launches a 3 µs kernel every 10 µs, buying a faster GPU will not fix it.

Walking through the layers

From the bottom up:

1. Hardware — GPU (or other accelerator), HBM capacity and bandwidth, NVLink or PCIe, multi-node fabric. See GPU architecture.
2. Kernels — CUDA, Triton, cuBLAS, FlashAttention, FlashInfer: the code that touches tensor cores and HBM. See kernels.
3. Model runtime — how the graph runs: PyTorch eager, torch.compile, TensorRT, custom engines.
4. Serving engine — vLLM, SGLang, TensorRT-LLM and similar: continuous batching, paged KV, scheduling, OpenAI-style APIs. See serving engines.
5. Router / gateway — auth, rate limits, model routing, KV-aware balancing, fallbacks. See routing.
6. Autoscaler / fleet — replicas, cold starts, packing LoRAs, capacity planning. See autoscaling.

A change at one layer can unlock or waste another. Quantizing weights without enough batch to become compute-bound leaves you bandwidth-bound with smaller reads. Continuous batching without PagedAttention fragments GPU memory until you cannot fill the batch.

The big idea

A restaurant has a stove, a cook, a ticket rail and a host. If tickets pile up at the host, hiring a hotter stove does nothing. The stack is that kitchen. Name the layer before you change a flag.

The analogy breaks when two layers couple: a gateway that ignores prefix cache sends work to a cold replica, and the engine looks "slow" when the router was wrong.

Common mistakes

• Blaming the GPU for host-side launch overhead or a saturated gateway queue.
• Tuning engine flags when the router sprayed cache-hot prompts across every replica.
• Quantizing then unpacking to a wider dtype in the runtime, so the kernel layer never sees the savings.

Check yourself

1. A Nsight timeline shows tiny gaps between short kernels and idle tensor cores. Which layer? Answer: overhead at the runtime or kernel-launch layer, not peak FLOPS.
2. Prefill is fast on one replica and slow on another for the same prompt. Where do you look first? Answer: the gateway and prefix-cache locality, not the model weights.

Key takeaways

• The stack is hardware → kernels → runtime → engine → gateway → autoscaler.
• Debug the layer that owns the symptom.
• A win at one layer can be wasted by a miss at another.
• Name the layer in the incident before you change silicon or flags.

Further learning

Next, walk one request through those layers on what inference engineering is.