Case study: designing an LLM serving platform

You have the pieces: engines, KV math, batching, parallelism, routing, autoscaling and cost models. This capstone forces them into one design. The brief: serve an open 70B-class base model plus a fleet of tenant fine-tunes to a peak of ~50k requests per minute, under interactive SLOs, at a target cost you can defend to finance. Think of it as the inference half of an LLM chat product.

Numbers below are worked assumptions for teaching. Re-benchmark on your GPUs before you buy anything.

Interactive SLOs: p90 TTFT < 800 ms, p90 TPOT < 40 ms (~25 tok/s/user minimum feel), error budget < 0.1% 5xx after retries exclude client cancels. Peak 50,000 requests/min ≈ 833 req/s. Multi-tenant LoRA: dozens to hundreds of adapters on shared bases (multi-LoRA). Cost target: plug your X from cost per token.

A credible design states: SLOs and workload mix with measured tokens/request; engine + FP8 + TP width + LoRA plan; goodput from a concurrency sweep → GPU count + $; gateway quotas and prefix-aware routing; autoscaling signals, warm pool size, batch preemption; canary/eval gates and node-loss runbooks. That is the difference between "we installed vLLM" and "we run inference as a product."

When someone asks you to design ChatGPT infra, lead with SLOs and tok/s, not with Kubernetes YAML. State the goodput assumption, show the replica count, name the scale signals, and call out cold-start lead time.

Do not parallelize "buy 130 GPUs" before the concurrency sweep. Hardware follows measured goodput. A realistic build is weeks: benchmark and FP8 eval, then single-region chat plus LoRA, then gateway quotas and prefix routing, then autoscaling and a warm pool, then the batch tier and cost dashboards, then multi-AZ drills.

Concepts in this section

Default interactive stack: FP8 + modest TP + continuous batching + prefix cache, plus speculative decoding when acceptance rates improve tok/s within the TPOT SLO. Pick vLLM or SGLang from team familiarity, LoRA maturity and prefix-cache features. Enable chunked prefill for long RAG.

Common mistakes

• Buying GPUs from a blog's single-stream tok/s instead of SLO-constrained goodput on your length mix.
• One pool for chat and batch with no preemption.
• Scaling on GPU util while TTFT burns in the waiting queue.
• Floating latest tags for checkpoints and LoRAs.
• No published overload policy, so 429s look like "flaky GPUs."

Key takeaways

• Convert RPM → tok/s → replicas using SLO-constrained goodput, not peak benchmark tok/s.
• FP8 + modest TP + continuous batching + prefix cache is the default 70B interactive stack.
• Gateway does quotas, priorities and cache-aware routing. Autoscaler watches queue, KV and TTFT.
• Batch on spare or spot with preemption funds utilization without stealing interactive SLOs.
• Observe tokens and $ per tenant. Canary models with evals. Drain and replace on GPU death.
• Re-benchmark before procurement. Capacity math is only as good as the concurrency sweep.

Further learning

vLLM Production Stack office hours — Kubernetes deployment, prefix-aware routing and KV sharing:

LLM Inference Performance Engineering: Best Practices (Databricks)databricks.com vLLM documentationdocs.vllm.ai NVIDIA DynamoA Datacenter Scale Distributed Inference Serving Framework - ai-dynamo/dynamoGitHub