What inference engineering is

Training a frontier model is a one-time (or infrequent) capital event. Serving it is a forever operating cost. For any model that sees real traffic, lifetime inference compute dwarfs training compute — often by an order of magnitude or more. A few percent more tokens per second per GPU compounds into millions of dollars a year at hyperscale, and into whether a product can afford to call a 70B-class model at all.

Inference engineering is the craft of turning a trained checkpoint into a service that is fast enough for users, cheap enough for the business, and reliable enough for production. It sits between ML research and classical SRE: you need enough of the model to know what the GPU is doing, and enough of systems to know what the fleet is doing.

You may never write a CUDA kernel and still make these decisions: API vs self-host, model size, streaming, prompt cache, SLO. The vocabulary below is how they get justified in a design review.

A request's life

Walk one chat completion through the stack to see where engineering time goes.

1. Gateway authenticates, applies token-bucket rate limits, and picks a model replica (maybe sticky to a prefix cache).
2. Scheduler admits the request into a continuous batch when there is KV space and SLO headroom — or queues it.
3. Prefill runs the prompt through every layer, writing K/V into the cache. Long prompts dominate TTFT.
4. Decode loops: one forward per new token, streaming tokens back over HTTP/SSE.
5. Completion frees KV pages (unless prefix caching keeps them), records metrics, bills tokens.

Failures map onto layers: OOM in step 3 is memory planning; high TPOT in step 4 is bandwidth, batching or kernels; unfair latency is scheduling; surprise bills are economics. Naming the layer keeps incident response short.

Prefill is parallel and often compute-bound. Decode is one token at a time against a growing KV cache, often memory-bound. Chat UX cares about both. Product SLOs are TTFT and TPOT percentiles, not FLOPS. Goodput is throughput that still meets those SLOs. Full treatment in inference metrics.

Concepts in this section

Then read GPU architecture and the roofline. For the host-or-buy decision, see API vs self-host. Self-host when volume is steady and constraints force it. Use an API when traffic is spiky, ops cost dominates, or you need a closed model. Hybrid is common.

Common mistakes

• Optimizing the wrong phase. Speeding up decode kernels when TTFT is the complaint (prefill or queueing) wastes a week.
• Benchmark theater. Publishing tokens/s at infinite concurrency with no TTFT cap.
• Ignoring KV. A model that "fits" in 80 GB at batch 1 OOMs at batch 32 because KV grew with concurrency and context.
• API complacency. Assuming provider latency is free of your prompt-cache and rate-limit choices.
• Heroic self-hosting. One GPU box without metrics, continuous batching or paging — then declaring open models "too slow."

Key takeaways

• Inference dominates lifetime compute for popular models. Small efficiency gains matter at scale.
• Latency, throughput and cost trade off. Batch size is the main dial.
• Debug the layer that owns the symptom, from GPU up to the gateway.
• Track TTFT, TPOT, goodput and $/M tokens. Peak tokens/s without SLOs is not capacity.
• Self-host when utilization and constraints justify ops cost. Otherwise APIs or inference providers win.

Further learning

Horace He's essay is the best short introduction to the three performance regimes (compute, bandwidth, overhead).

Making Deep Learning Go Brrrr From First Principles (Horace He)Making Deep Learning go Brrrr From First Principleshorace.io

Databricks' best-practices post maps the same ideas onto LLM serving metrics.

LLM Inference Performance Engineering: Best Practices (Databricks)databricks.com

The JAX scaling book's inference chapter is the napkin-math companion.

How to Scale Your Model: All About Transformer InferencePerforming inference on a Transformer can be very different from training. Partly this is because inference adds a new factor to consider: latency. In this section, we will go all the way from sampling a single new to…How To Scale Your Model

Horace He's Jane Street talk is a spoken version of the same first-principles story.