Inference engineering

Sections: 6
Pages: 138
Reading time: ~5 h
Take first: ML, Infra & SRE

What you will be able to do

Explain GPU architecture and use the roofline model to predict whether a kernel or workload is compute- or memory-bound.
Compute, on a napkin, the memory footprint, KV cache size, latency floor and throughput ceiling of serving a given model on given GPUs.
Explain and configure continuous batching, PagedAttention, FlashAttention, quantization, speculative decoding and prefix caching, and predict their effect on TTFT, TPOT and throughput.
Choose tensor, pipeline, expert and data parallelism and prefill/decode disaggregation for a model and workload.
Deploy and tune vLLM or SGLang, and benchmark it correctly across a concurrency sweep.
Build a cost model from $/GPU-hour to $/million tokens and decide between APIs and self-hosting.
Design an autoscaled, multi-model inference platform with KV-aware routing and SLO-based capacity planning.

Most of the money and GPUs spent on a popular model after launch go to inference: answering user requests, not training the weights. A naive deployment — one request at a time, FP16 weights, no continuous batching — can easily be 10× more expensive per token than a tuned stack on the same hardware. The difference is not a single trick. It is a chain of decisions: which GPU, how memory is laid out, whether the workload is compute-bound or memory-bound, how you batch, how you cache KV, how you quantize, and how you route and autoscale a fleet.

This topic is the engineering bible for that chain. It starts with GPU architecture and the roofline model so you can predict performance before you benchmark. It then builds the arithmetic of LLM serving — prefill vs decode, KV cache size, latency floors — and the optimizations every modern engine relies on. It closes with engines (vLLM, SGLang, TensorRT-LLM), fleets, and the economics of $/million tokens.

Who this is for

You are a software engineer who needs to serve models — or to argue intelligently with people who do. You are comfortable with Python and systems ideas (latency, memory, concurrency). You do not need to have written CUDA before; the early lessons teach the mental model from first principles.

If you mostly call APIs and care about prompts, RAG and agents, start with AI engineering. If you want how transformers are trained, see ML. This topic sits on the serving and systems side: GPUs, kernels, engines and cost.

How to study it

Read the GPU fundamentals unit carefully once. Everything later — why decode hates batch-1, why FlashAttention exists, why continuous batching wins — is a consequence of bandwidth, occupancy and arithmetic intensity.

For each deep lesson, do one napkin calculation with a real model and GPU (a 7B or 70B checkpoint on an H100-class card is enough). Then, when you have access to a GPU, run a small serving engine and measure TTFT and tokens/s under a concurrency sweep. The Further learning links are depth on demand, not a reading list to finish first.

138 lessons, about 5 hours of reading. Treat units as modules you can revisit when a production decision appears.

Hands-on loop that sticks: pick one open model, serve it with vLLM or SGLang, plot TTFT and TPOT vs concurrency, then change one knob (max batch, dtype, tensor parallel) and predict the direction of the curves before you look. Prediction-plus-measurement beats passive reading.

Projects worth doing once: (1) a spreadsheet cost model from $/GPU-hour to $/M tokens at several utilizations; (2) a KV-cache size calculator for your context lengths; (3) a flame/timeline capture of one decode step explaining whether you are compute-, memory- or overhead-bound.

Unit roadmap

The numbered list is a shorter path. The sections below are every lesson in the topic.

GPU fundamentals

Hardware and the performance model that makes the rest of these pages predictable.

What inference engineering is — latency–throughput–cost, the serving stack, when not to self-host
GPU architecture — SMs, warps, tensor cores, HBM and interconnects with H100/B200 numbers
The CUDA programming model — grids, blocks, threads, coalescing, occupancy, Triton
The roofline model and arithmetic intensity — compute-bound vs memory-bound, with worked examples
Kernels, fusion and compilation — fusion, torch.compile, CUDA graphs, kernel libraries

Anatomy of LLM inference

The math of a single request and of a batch.

Inference arithmetic — memory, FLOPs/token, latency floors and throughput ceilings
Prefill vs decode — why one phase is compute-bound and the other memory-bound
The KV cache — footprint, GQA/MLA, quantization and offloading
Inference metrics — TTFT, TPOT, throughput, goodput

Core optimizations

What every serious serving engine does.

Scaling out

One GPU is not enough for large models or high QPS.

Serving systems

Engines, measurement and gateways.

Production and economics

Fleets, money and everything that is not a chat LLM.

Course resources

Horace He's first-principles essay is the fastest way to internalize the three performance regimes (compute, memory bandwidth, overhead) that this topic uses everywhere.

Making Deep Learning Go Brrrr From First Principles (Horace He)Making Deep Learning go Brrrr From First Principleshorace.io

The JAX scaling book chapters on GPUs, rooflines and inference are the best free long-form reference for napkin math.

How to Scale Your Model: All About Transformer InferencePerforming inference on a Transformer can be very different from training. Partly this is because inference adds a new factor to consider: latency. In this section, we will go all the way from sampling a single new to…How To Scale Your Model

Horace He's Jane Street talk expands the same ideas into compilers, fusion and CUDA graphs — worth watching before the kernels lesson.

Horace He: Building Machine Learning Systems for a Trillion Trillion Floating Point OperationsJane Street

For CUDA itself, NVIDIA's modern CUDA C++ class is a clean on-ramp if you want to write a kernel after reading the programming-model lesson.

Accelerating Applications with Parallel Algorithms | CUDA C++ Class Part 1NVIDIA Developer

GPU fundamentals

The latency-throughput-cost triangle

Every serving choice trades user wait, tokens per GPU, and dollars per million tokens. Batch size is the main dial.

The inference stack

Serving is layers from GPU to gateway. Wins and outages live at one layer. Blaming the GPU for a Python launch tax is a category error.

What inference engineering is

Turning a checkpoint into a fast, cheap, reliable service. Start here for the latency-throughput-cost map and the serving stack.

Warps and SIMT

GPUs schedule work in warps of 32 threads that share one instruction. Divergence and occupancy decide how much of the chip you actually use.

GPU memory hierarchy

Registers, shared memory, L2 and HBM, with H100 numbers, and why inference runs out of memory more often than it runs out of FLOPs.

GPU architecture

SMs, warps, tensor cores and the memory hierarchy from registers to HBM, with H100/B200 numbers.

Tensor cores and peak FLOPs

Tensor cores do the matrix math that LLMs need. Use the dense peak, then the ridge point, to see whether a kernel can ever hit that number.

GPU interconnects

NVLink, PCIe and InfiniBand set how fast GPUs can share work. Keep latency-sensitive collectives inside an NVLink domain.

Kernels, grids and threads

A kernel is a device function launched as a grid of blocks of threads, each computing a different index.

Coalescing and shared-memory tiling

Adjacent warp loads become few HBM transactions, and tiles in shared memory let you reuse each loaded byte.

The CUDA programming model

Kernels, grids, blocks and threads, memory coalescing and occupancy, plus Triton as the higher-level path.

Occupancy and launch overhead

Resident warps hide HBM latency, and a 5–10 µs launch tax wrecks decode when every op is tiny.

Triton and when to write a kernel

Prefer library GEMMs and attention. Use Triton for fusions. Drop to raw CUDA only when you need control the compiler cannot express.

Arithmetic intensity and the ridge point

Intensity is FLOPs per byte moved from HBM. The ridge is where the bandwidth slope meets the compute roof.

Roofline examples

Elementwise add sits far left of the ridge. A large square matmul sits to the right. Tiny GEMMs fall back onto the bandwidth slope.

The roofline model and arithmetic intensity

Predicting performance from FLOPs and bytes: compute-bound vs memory-bound. Start here for the map, then work the examples.

Decode batch size on the roofline

Decode intensity scales with how many sequences share one weight read. Batch-1 crawls the bandwidth slope. Batching walks you toward the roof.

Communication rooflines

Multi-GPU serving has a second roof. When collective time exceeds local compute, more tensor-parallel GPUs can slow tokens/s.

Launch overhead

Decode runs many short kernels. Python dispatch and kernel launches can cost a millisecond per token before any matmul starts.

Kernel fusion

Combine several ops into one GPU kernel so intermediates stay in registers or shared memory instead of round-tripping through HBM.

Kernels, fusion and compilation

Kernel fusion, torch.compile, CUDA graphs and the libraries that make models fast.

CUDA graphs

Record a fixed sequence of GPU launches once and replay it every token, so the CPU is no longer the metronome for decode.

torch.compile

PyTorch traces a model into a graph, fuses the glue with Inductor and Triton, and can wrap the result in CUDA graphs. Dynamic shapes are the catch.

Anatomy of LLM inference

Inference arithmetic

Napkin math for any model on any GPU: weight memory, decode and prefill time, batch limits and KV budget.

Weight memory

How many bytes a model's weights take, from parameter count times bytes per parameter, and why precision decides what fits.

Decode latency floor

Why one generated token costs roughly the weight bytes divided by memory bandwidth, and how that gives a tokens-per-second ceiling.

Prefill time

How long the prompt takes to process before the first token, estimated from parameter count, prompt length and achieved FLOPs.

Batch ceiling

How batching raises decode throughput until compute runs out or the KV cache fills, and how to estimate the concurrency limit.

The prefill phase

Prefill reads the whole prompt in parallel, writes the KV cache, and emits the first token. It is usually compute-bound and sets time to first token.

The decode phase

Decode emits one token per step, re-reads the weights and the growing KV cache, and is usually memory-bandwidth-bound. It sets time per output token.

Prefill vs decode

The two phases of LLM inference. Prefill is compute-bound and sets TTFT. Decode is memory-bound and sets TPOT. They interfere when they share a GPU.

Inference workload shapes

Chat, RAG, reasoning and agents sit in different corners of the input-versus-output plane. Each corner wants a different capacity plan.

Why the KV cache exists

Each decode step would rebuild keys and values for every past token. The cache stores them once and appends one new column per step.

KV bytes per token

Each token stores K and V on every layer. Llama-3-70B in BF16 is about 320 KB per token, so a long chat can rival a small model.

The KV cache

What keys and values the cache stores, how big it gets per token, and how GQA, MLA, quantization and offloading shrink it.

GQA, MQA and MLA

Sharing or compressing KV heads shrinks the cache and the bytes moved each decode step, which is usually a better first fix than quantizing a bloated MHA cache.

KV offloading

Hot tokens stay in HBM. Warm prefixes spill to CPU or SSD. Hits beat recomputing prefill. Cold million-token decode still pays the bandwidth.

TTFT and TPOT

Time to first token is queue plus prefill. Time per output token is decode smoothness. End-to-end latency is those two put together.

Goodput

Throughput is tokens you completed. Goodput is tokens that also met the latency contract. Capacity planning uses the second number.

Inference metrics

TTFT, TPOT/ITL, end-to-end latency, throughput and goodput — definitions, percentiles and how they trade off as concurrency rises.

The latency-throughput curve

As concurrency rises, tokens per second climb then flatten, per-user speed falls, and tail TTFT spikes. The SLO knee is the operating point.

Inference SLOs

Write dual thresholds for TTFT and TPOT, name the workload next to them, and autoscale on goodput headroom rather than GPU utilization.

Core optimizations

Why batching speeds up decode

One weight read from HBM can serve many sequences, so larger batches raise arithmetic intensity.

Static and dynamic batching

Padding and tail idle waste GPU slots, and a timeout still locks the batch until the longest sequence ends.

Batching and continuous batching

Why batching amortizes weight reads, how static and dynamic batching waste GPU time, and how iteration-level continuous batching plus chunked prefill unlocks 10–20× throughput.

Continuous batching

Reschedule every model forward so finished sequences free a slot on the next step, not when the longest one ends.

Chunked prefill

Split a long prompt into token chunks so decode sequences keep making progress every iteration.

Why reserved KV memory wastes the GPU

Reserving one contiguous block of KV cache for the longest allowed sequence leaves most of it empty, and scatters the free space into pieces too small to use.

PagedAttention and KV memory management

How reserving contiguous max-length KV wastes most of GPU memory, and how fixed-size KV blocks with block tables eliminate fragmentation and unlock sharing.

Block tables and KV pages

PagedAttention cuts the KV cache into fixed-size blocks and maps each sequence's tokens to them with a block table. Blocks are allocated only as a sequence grows.

Copy-on-write KV sharing

Sequences that share a prompt can point at the same KV blocks. Only the block that diverges is copied, which makes parallel sampling and beam search cheap.

Preemption, swap and recomputation

When KV memory runs out under load, the scheduler pauses some sequences. It either swaps their blocks to host memory or drops them and recomputes the prefill later.

Why attention is HBM-bound

Standard attention writes an n by n score matrix to HBM. Bytes grow with n squared, so long sequences leave tensor cores idle.

Online softmax

Softmax needs a row max and a sum. Online softmax keeps running statistics so you can process the row in tiles and still get the exact result.

FlashAttention

Why standard attention is HBM-bound, how tiling plus online softmax computes exact attention in on-chip SRAM, and what FlashAttention-2/3 and decode variants change.

FlashDecoding and split-KV

Decode often has one query and a long KV cache. Split the KV across thread blocks, then reduce, so the GPU has parallel work again.

FlashAttention-2 and 3

Later FlashAttention kernels keep the same math. FA2 improves work partitioning. FA3 targets Hopper async copies and FP8.

Quantization formats and scales

How many bits a weight uses, and how scales stretch a coarse grid over a tensor. Per-tensor, per-channel and per-group scales trade overhead for quality.

Quantization

INT8, FP8, INT4 and FP4. Fewer bits shrink weights, activations and KV. Match the format to memory-bound decode or compute-bound prefill, then eval quality.

Weight-only vs activation quantization

W4A16 stores 4-bit weights and keeps 16-bit activations. W8A8 and FP8 quantize both sides so Tensor Cores run native low-precision matmuls.

Activation outliers

A few channels with huge magnitudes set a single INT8 scale and squash everyone else. LLM.int8() and SmoothQuant exist to stop that.

GPTQ and AWQ

Two post-training methods that compress a finished checkpoint to INT4 with a small calibration set. GPTQ uses second-order updates. AWQ protects salient channels.

KV cache quantization

Weights are only half the story at long context. Quantizing KV to FP8 or INT8 or INT4 shrinks the cache, raising batch size or max context.

Draft and verify

A cheap draft proposes k tokens. The target scores them in one forward. Rejection sampling keeps the target's distribution.

Speculative decoding

Draft-and-verify so one target forward accepts several tokens. Same output distribution. Real speedups when decode is memory-bound and acceptance stays high.

Speculative acceptance rate

Speedup is expected accepted tokens divided by draft-plus-verify cost. Sweep k. Measure alpha on your traffic.

Draft sources

Small LMs, prompt n-grams, Medusa heads, EAGLE and multi-token predictors. Pick for stack support and tokenizer match.

Speculation versus batching

Speculation is a latency tool for underfilled GPUs. At high batch it can steal compute from everyone in the continuous batch.

Shared prompt prefixes

System prompts, tools, few-shots and chat history repeat across requests. Those shared heads are the tokens prefix caching can skip.

Prefix caching

Reusing KV across requests that share prompts. Block hashing, RadixAttention, cache-aware routing, and why cached input tokens are billed cheaply.

Automatic prefix caching

With paged KV, each block is hashed from its tokens. Identical spans reuse the same physical blocks, so prefill can skip work without an application change.

RadixAttention

SGLang stores retained KV in a radix tree keyed by token sequences. Shared prefixes are shared paths from the root, with LRU eviction at the leaves.

Cache-aware routing

Prefix cache lives on one replica. A load balancer that ignores cache state sprays requests and destroys hit rate. Send each request toward the replica that already holds its prefix.

Multi-tier prefix cache

Extend a radix or block cache from GPU HBM to host DRAM and then SSD. Larger working sets, slower hits. Worth it for long-lived sessions, not short one-offs.

Scaling out

Parallelism for inference

Tensor, pipeline, data and context parallelism: splitting a model across GPUs so weights and KV fit, and single-request latency stays under SLO.

Tensor parallelism

Split each layer across GPUs so every token uses every rank. Column-parallel and row-parallel matmuls plus all-reduces do the work.

Pipeline parallelism

Assign blocks of layers to different GPUs or nodes. Inter-stage traffic is small, but decode leaves pipeline bubbles that waste GPU time.

Data parallelism for inference

Independent replicas of an already-sharded model. Each replica has its own KV pool. Throughput scales; single-request latency does not.

Total vs active parameters

A mixture-of-experts model stores far more weights than it uses for each token. Memory follows the total. Compute follows the active count.

Serving mixture-of-experts models

Expert parallelism, all-to-all dispatch/combine, and load balancing for sparse MoEs like DeepSeek-V3 — where memory tracks total params but FLOPs track active ones.

Expert parallelism and all-to-all

Expert parallelism puts different experts on different GPUs. Each MoE layer sends tokens to their experts and back with two all-to-all steps.

Hot experts and load imbalance

Routers rarely spread tokens evenly. One hot expert can overload its GPU, and because the combine step waits for everyone, the whole layer slows down.

Different EP widths for prefill and decode

Prefill and decode send very different numbers of tokens to each expert, so the best expert-parallel width is usually different for each phase.

Prefill–decode interference

A long prefill on the same GPU as decode steals SMs and HBM bandwidth, so everyone else's TPOT spikes.

KV transfer

Disaggregation ships the KV cache to a decode worker. Multi-GB cold prompts on slow links blow the TTFT budget.

Disaggregated prefill and decode

Running prefill and decode on separate GPU pools so each phase gets its own parallelism, batching, and hardware — and when colocated chunked prefill is enough.

Sizing prefill and decode pools

Set GPU counts from prefill GPU-seconds and decode GPU-seconds per request, not a 50/50 split.

Disaggregation failure modes

Orphan KV, slow-start decode, lost cache affinity and version skew are the outages you do not have when pools are colocated.

Why long context is expensive

Prefill attention grows with n squared. KV and decode reads grow with n. A million-token prompt is not eight times a 128k prompt.

Context parallelism

Shard the sequence across GPUs and pass KV or Q around a ring. This cuts long prefill latency. On decode it often hurts.

Serving long contexts

Million-token prompts: quadratic prefill attention, linear KV growth, context parallelism, offloading, sparse/hybrid architectures, and how providers price the pain.

Sliding-window and hybrid attention

Local windows cap KV growth. Hybrids mix local layers with periodic global attention. Prefer them when the product does not need lossless full attention.

Serving systems

Anatomy of a serving engine

API, scheduler, KV manager, executor, sampler, detokenizer. One forward pass packs every sequence the scheduler admitted.

Serving engines: vLLM, SGLang and TensorRT-LLM

How modern engines are built, how they differ, and which knobs move TTFT, TPOT and tokens per GPU-second.

vLLM

The broad default engine. PagedAttention, a unified iteration-level scheduler, and wide model coverage.

SGLang

Built for prefix-heavy and structured workloads. RadixAttention shares KV the way a filesystem shares path prefixes.

TensorRT-LLM

Compile a model into a GPU-specific engine. You pay a rebuild cost and gain tightly tuned NVIDIA kernels.

Engine knobs that matter

Tensor parallel size, max model length, GPU memory utilization, quantization, speculation and prefix caching. Ignore the long flag list until these are right.

Goodput and realistic workloads

Capacity is throughput that still meets the SLO, measured on your length mix, not on 512/128 prompts.

Closed-loop vs open-loop load

Fixed concurrency finds the latency–throughput curve. An arrival process shows what happens when QPS exceeds goodput.

Benchmarking inference

Measuring honestly: workloads, load generators, concurrency sweeps and the mistakes that make numbers meaningless.

Concurrency sweeps

Warm up, step concurrency, plot latency against throughput, and take the highest point that still meets the SLO.

Benchmark pitfalls

Cache hits, guessed tokens, client bottlenecks and peak-only slides are how inference numbers become marketing.

Why round robin fails for LLMs

Two HTTP requests can differ by 100× in GPU-seconds. Equal request counts do not equal equal load when cost, cache and streams are uneven.

Replica scoring

Beat round robin with an explicit score from prefix affinity, queue depth, KV headroom and adapter locality. Hard filters first, then pick the max.

Routing, load balancing and scheduling

LLM-aware gateways: KV-cache-aware routing, token-based rate limits, priorities and fairness.

Token-based rate limits and gateway policy

Limit estimated tokens, not request counts. Add priorities, fair queues, cancellation and hard max_tokens at the gateway.

Admission control and load shedding

When offered load exceeds goodput, queue with a deadline, shed batch first, then reject with Retry-After. Do not OOM the replica.

LoRA adapters

LoRA fine-tunes a model by training a small add-on beside frozen weights. The add-on is tens of megabytes, so one base model can carry thousands of them.

Multi-LoRA and multi-model serving

Serving many fine-tuned adapters and models on shared GPUs efficiently.

Multi-LoRA batching and adapter paging

One base model can serve thousands of LoRA adapters if one forward pass handles requests for different adapters, and cold adapters are paged in from host memory.

GPU sharing with MIG, MPS and time-slicing

MIG, MPS and time-slicing let several workloads share one GPU. Each trades isolation against density, and each fits different workloads.

Tenant isolation and noisy neighbors

When tenants share GPUs, one heavy customer can push everyone else's latency up. Admission caps, fair scheduling, priority classes, separate pools and per-request limits keep tenants apart.