Guide

Production and economics

API vs self-host

Self-hosting wins when GPUs stay busy on a model you accept. APIs win when volume is spiky or frontier quality matters more than cents.

123 of 138 · 2 min read

Self-hosting wins when you can keep GPUs busy on a model quality you accept. APIs win when volume is spiky, engineering time is scarce, or the frontier model quality gap matters more than cents. The break-even is not a peak-benchmark $/M. It is fully loaded cost at the utilization you will actually run.

The big idea

Owning a bakery is cheaper per loaf only if you bake every day. Buying bread by the slice is cheaper if you eat two a week. The analogy breaks when the "bread" is a closed frontier model you cannot bake at all — then self-hosting an open 70B is a different product, not a cost optimization.

Walking through the checklist

  1. Volume. At low QPS, the fixed cost of even one reserved GPU dominates. APIs' marginal pricing is kinder.
  2. Utilization. Self-host at 20% utilization often loses to APIs even if peak $/M looks great. See the token cost formula.
  3. Engineering + ops. On-call, upgrades, CUDA drivers, eval regressions — real OpEx. Fold a person-month into the model if you are honest.
  4. Quality gap. If the product needs a closed frontier model, "self-host an open 70B" is a different product.
  5. Data / compliance. Air-gapped or "weights never leave our VPC" can force self-host regardless of price.

A napkin: if an API charges $0.50 / M output and your fully loaded self-host (GPU + eng amortization) is $0.30 / M at 70% util, self-host wins on cost — until util falls to 35% and you are back at $0.60 / M.

Trade-off

Choose API when traffic is bursty, you need frontier quality, or the team is small. Choose self-host when you have steady multi-thousand tok/s demand, an open model that passes evals, and staff who can run serving engines. Hybrid is common: frontier API for hard cases, self-host for the high-volume easy path.

Common mistakes

  • Comparing API list price to peak self-host $/M. Use the utilization you will actually run.
  • Ignoring eng and on-call. A cheap GPU-hour with no owners is not cheap.
  • Self-hosting to "save money" when you need the frontier model. You changed the product.

Check yourself

  1. Self-host is $0.30 / M at 70% util. What is it at 35% util, same GPU bill? Answer: about $0.60 / M, because tokens/hour halved.
  2. When can compliance force self-host even if the API is cheaper? Answer: when weights or prompts must not leave your VPC.

Key takeaways

  • Break-even hinges on volume, utilization, eng cost and quality — not peak tok/s.
  • Half the utilization, roughly twice the self-host $/M.
  • Hybrid (frontier API + self-host for the easy path) is a normal design, not a cop-out.

Further learning

Next, put a price on the product in pricing inference products.

The economics of inference

Turn $/GPU-hour into $/million tokens with utilization and throughput, then decide API vs self-host and price products with real margins.

The token cost formula

$/M tokens is pool dollars per hour divided by sustained tokens per second. Utilization is the silent multiplier.

Serving engines: vLLM, SGLang and TensorRT-LLM

How modern engines are built, how they differ, and which knobs move TTFT, TPOT and tokens per GPU-second.