Case study: LLM chat product

An LLM chat product is a real-time distributed system wrapped around a scarce accelerator pool. Users expect the first words within about a second, memory across turns, file uploads and tools. You pay per token, and at peak you run out of GPUs long before you run out of web servers.

The target for this design: time to first token (TTFT) under about 1 second, conversation history across sessions, file uploads and tools, roughly 10 million daily users, tiered quotas, and graceful degradation when capacity runs short.

A useful order for the design is query, then model, then response. Add the gateway, routing, retrieval and tools, guardrails, caching and orchestration after that. This case study covers the system design and the production skeleton, not how to train the model.

Request path

1. API gateway: TLS, a web application firewall, request size limits and edge rate limits.
2. Auth and quotas: the session or API key, and the remaining token budget for the current window.
3. Orchestrator: loads the conversation, assembles the prompt, calls tools or retrieval, invokes the model, streams the answer back and saves the assistant message.
4. Model router: picks a model by tier, latency target, capability and current load.
5. Inference fleet: self-hosted GPUs, or vendor APIs behind one gateway.

Streaming needs the proxies to cooperate. Use server-sent events or chunked HTTP, and turn off response buffering in nginx or Envoy. A proxy that buffers the whole completion ruins time to first token.

Concepts in this section

Common mistakes

• Measuring only total latency. Users judge the wait for the first token. Track TTFT separately.
• Limiting by requests, not tokens. One long request can cost more than a hundred short ones.
• Buffering streams in proxies. The user sees nothing until the whole answer is ready.
• Resending whole files every turn. Keep the file in storage and attach a reference.
• Letting a free tier starve paying users. Use priority and tier-aware admission.
• No record of which model and prompt produced a reply. Store the prompt version, model ID and router reason on every turn.

Key takeaways

• The path runs gateway, auth, orchestrator, router and fleet, with tokens streamed back.
• Size the fleet from tokens per day and peak tokens per second, then add headroom.
• Store the conversation as turns and context references, and assemble a bounded prompt each time.
• Limit by tokens, shed load by tier, and degrade gracefully instead of failing.
• Treat tool output and uploads as untrusted input, and make every tool call bounded.

Further learning

Chip Huyen's writing on building LLM platforms covers the gateway, routing and evaluation layers in more depth.

Chip Huyen: building LLM applicationsI work to bring AI into production. I write about AI system design.Chip Huyen

Andrej Karpathy's talk on how large language models work gives the background for tokens, context and sampling.

For the serving side, see LLM APIs and the inference serving platform.