Latency and cost optimization

An LLM request is not a database query. You pay for every input and output token, and wall-clock time is dominated by how many tokens you send, how many you generate, and which model does the work. A support bot that feels snappy at ten users can burn a six-figure monthly bill at a million conversations if you never measure cost per successful task.

Latency and cost are the same knobs once you treat quality as a hard constraint. Optimize accuracy first. A cascade that deflects the wrong tickets destroys CSAT faster than it saves API spend. Gate every cost change on your eval suite.

A support bot at 1M conversations per month

Assumptions for planning, not a quote: 1,000,000 conversations per month, 3 model turns each, mid-tier $3 / M input and $15 / M output. Baseline: 3,500 input tokens plus 300 output tokens per turn, no cache, no routing.

Turns/month   = 1e6 × 3 = 3,000,000
Input tokens  = 3e6 × 3,500 = 10.5e9    →  $31,500
Output tokens = 3e6 × 300   = 0.9e9     →  $13,500
Total         ≈ $45,000 / month         →  $0.045 / convo

Cut input to 1,400 and output to 180 (RAG plus shorter answers) and the bill drops to about $20,700 (−54%). Add prompt caching at an 80% hit rate on 1,000 of those 1,400 tokens, with cached reads at ~10% of base input, and you land near $14,220 (−68%). Route 75% of turns to a small model at about $0.40 / $1.60 per million and the blend can approach $5,500 (−88%), or about $0.0055 per conversation.

The structure matters more than the exact discount: tokens × price tier × cache hit rate. A human support interaction often costs several dollars fully loaded. At roughly $0.01 to $0.05 model cost you still have room for tooling and gross margin, but only if you keep measuring.

Concepts in this section

For prompt-prefix reuse, see caching. For GPU-side math, see cost per token. Offline jobs belong on batch LLM APIs, often about 50% off sync prices as of when this was written.

Walk this list, in order

1. Is quality good enough on a fixed eval? If not, stop.
2. What does the user wait on: first token, full answer, or job completion?
3. Can you stream or show progress without changing the model?
4. Can you remove tokens without hurting the eval?
5. Can independent work run in parallel?
6. Is there a detectable easy subset for a smaller model?
7. Is the traffic interactive (sync plus prompt cache) or offline (batch)?
8. Only then consider serving-side tricks or fine-tunes.

Most production wins stop at step 6.

Common mistakes

• Streaming TTFT while three serial tools run first. Users still wait. Move retrieval and account fetches ahead of generation.
• No max_tokens. A "short answer" prompt still lets a reasoning model emit thousands of tokens.
• Cascade ping-pong. If the small model always escalates, you pay for two calls. If escalate rate sits above about 40%, the router is wrong.
• Batch for interactive chat. You save money and destroy the product.

Key takeaways

• Instrument TTFT, tokens/s, end-to-end latency and cost per successful task.
• Stream for interactive UX. Cut tokens before exotic serving tricks.
• Route or cascade only when evals show a detectable easy/hard split.
• Quality gates every optimization.

Further learning

What We Learned from a Year of Building with LLMsA practical guide to building successful LLM products, covering the tactical, operational, and strategic.Applied LLMs

Claude docs: Reducing latencyReduce Claude's response latency by choosing a faster model like Claude Haiku 5.5, trimming prompt and output tokens, and streaming responses.Claude Platform Docs

OpenAI: Latency optimizationImprove latency across a wide variety of LLM-related use cases.OpenAI Developers