The token cost formula

A GPU costs roughly a fixed amount per hour whether it emits 100 or 10,000 tokens in that hour. Cost per token is the hourly burn divided by tokens produced. Every optimization that raises sustained tokens/s at fixed GPU count, or cuts GPUs at fixed tokens/s, drops $/token linearly.

The big idea

A factory that costs $2.50 an hour is cheap per widget only if it keeps shipping widgets. Half the rate, twice the unit cost. The analogy breaks when you care about input tokens and cached tokens too — those have different costs, covered next.

Walking through the arithmetic

For a steady serving pool:

$/M output tokens =
  ($/GPU-hour × GPUs_in_pool)
  / (sustained_output_tokens_per_second × 3,600)
  × 1,000,000

Equivalently: cost per second of the pool, divided by tokens per second, times a million.

Assumptions (illustrative, mid-2020s cloud rental ballpark — always re-quote):

• One H100 at $2.50 / GPU-hour
• Open ~8B model, FP8 or INT8-friendly serving
• Sustained 2,500 output tokens/s under a realistic concurrency mix

Tokens per hour = 2,500 × 3,600 = 9,000,000
$/M tokens      = $2.50 / 9     ≈ $0.278

Call it ~$0.28 / M output tokens at that utilization.

Suppose the peak goodput in a benchmark is 3,125 tok/s, and 2,500 was already 80% of peak. At 40% of peak you sustain only 0.4 × 3,125 = 1,250 tok/s:

Tokens per hour = 1,250 × 3,600 = 4,500,000
$/M             = $2.50 / 4.5   ≈ $0.56

Same GPU price, 2× the unit cost, because half the tokens leave the factory. This is why autoscaling and batching are finance problems, not only SRE problems.

Ten H100s at $2.50/h = $25/h. At 20,000 aggregate output tok/s: tokens/hour = 72M → ~$0.35 / M. Drop to 8,000 tok/s (idle afternoon plus over-provisioned SLO headroom): tokens/hour = 28.8M → ~$0.87 / M. The infra bill barely moved. The unit economics did.

If your cost model uses peak throughput and your ops run at half that, your real $/M is roughly 2× the spreadsheet.

Common mistakes

• Using peak benchmark tok/s in the denominator. Production often runs at 40–70% of that.
• Forgetting the 3,600. Tokens per second times hours, not tokens per second sitting naked under $/hour.
• Treating the GPU bill as the unit cost. The bill can stay flat while $/M doubles.

Check yourself

1. One GPU at $2.50/h sustains 2,500 output tok/s. What is $/M? Answer: 2,500 × 3,600 = 9M tokens/h, so $2.50 / 9 ≈ $0.28 / M.
2. Peak is 3,125 tok/s and you run at 40% of peak. What happens to $/M versus 80%? Answer: it doubles, from about $0.28 to about $0.56.

Key takeaways

• $/M = pool $/hour ÷ (tok/s × 3600) × 1e6.
• Utilization is half the battle. Peak tok/s in the spreadsheet is how you lie to yourself.
• Worked scale: ~8B on one H100 at $2.50/h and 2,500 tok/s ≈ $0.28 / M.

Further learning

Next, split the bill into input vs output.