Production and economics
The economics of inference
Turn $/GPU-hour into $/million tokens with utilization and throughput, then decide API vs self-host and price products with real margins.
Training spends make headlines. Inference spends the money. Once a model is in production, every user message burns GPU-seconds, and the gap between a naive deployment and a tuned one is routinely an order of magnitude in cost per token. Product pricing, "should we self-host?", and "is this feature affordable?" all collapse to the same arithmetic.
For application-level latency/cost tradeoffs see latency and cost. For the fleet that feeds utilization, see autoscaling.
Concepts in this section
Read them in order. Each one builds on the one before.
| Step | Concept | What it answers |
|---|---|---|
| 1 | The token cost formula | How do you go from $/GPU-hour to $/M tokens? |
| 2 | Input vs output cost | Why do prefill, cache hits and decode price differently? |
| 3 | API vs self-host | When does owning GPUs beat paying per token? |
| 4 | Pricing inference products | How do you set a price that keeps margin? |
A one-page cost model
Build this and revisit it after every major optimization or GPU price change:
- GPU SKU and $/h, GPUs per model
- Measured output tok/s at the concurrency you will actually run
- Utilization assumption (often 40–70% of peak)
- Separate lines for fresh input, cached input and output
- Compare to API list prices on the same workload mix
Worked scale from the formula page: ~8B on one H100 at $2.50/h and 2,500 tok/s ≈ $0.28 / M; at half util ≈ $0.56 / M.
The levers that move the number are the same ones in the rest of this topic: continuous batching, quantization, prefix caching and speculative decoding. Each should show up as higher sustained tok/s on the same GPU, not as a slide that quotes peak.
As of 2025–2026, three curves move at once: API token prices trend down, GPU rental swings with supply, and software efficiency (MLA, FP4, disaggregation) raises tokens per GPU. Treat any absolute $/M in a blog post as a snapshot. Recompute from your GPU quote and your sustained tok/s quarterly.
Walking through $/M = ($/h) / (tok/s × 3600) × 1e6 with a utilization caveat beats vague talk about "GPUs being expensive."
Common mistakes
- Peak tok/s in the denominator. Real $/M is often 2× the spreadsheet.
- One token counter. Prefill-heavy agents and decode-heavy chat need different fixes.
- Self-hosting on 20% util to beat an API. You will lose, then add on-call.
Key takeaways
$/M tokens = pool $/hour ÷ tokens per second × 1e6. Utilization is half the battle.- Meter fresh input, cached input and output separately.
- API vs self-host hinges on volume, utilization, eng cost and quality.
- Product-price from p95 COGS and cache-friendly design, or margin disappears at scale.
Further learning
SemiAnalysis × WEKA on token economics, interactivity vs $/token, and why prompt caching reshapes agent bills:
Artificial AnalysisComparison and analysis of AI models and API hosting providers. Independent benchmarks across key performance metrics including quality, price, output speed & latency.artificialanalysis.ai
Related
The token cost formula
$/M tokens is pool dollars per hour divided by sustained tokens per second. Utilization is the silent multiplier.
Gross margin and COGS, including AI
What belongs in COGS vs opex, why classic SaaS runs ~75–80% gross margin, why AI products often sit at 50–60%, and the levers that move the number.
Latency and cost optimization
Streaming, routing, cascades, batching and parallelism: making LLM apps fast and affordable without dropping quality.