Pricing inference products

If you sell an AI feature, gross margin is (revenue − inference COGS − other variable COGS) / revenue. Inference is often the dominant variable cost, unlike classic SaaS. Power users, retries, tool loops and "thinking" tokens define your margin risk. The mean user does not.

The big idea

A buffet priced for the average diner dies when a few people eat all afternoon. Meter usage and set prices, or limits, from p95 user cost. Architectures that raise cache hit rate (stable system prompts, shared RAG corpora) are margin features, not only latency features.

The buffet picture breaks when interactivity itself changes the cost. A product that needs 50–100 tok/s/user cannot pack the GPU the way a batched cheap tier can.

Walking through interactivity vs $/token

Providers sometimes publish a "cheap" token price that assumes heavy batching and low tokens/s/user. Coding and voice may need 50–100 tok/s/user. That burns decode bandwidth that cannot be shared as aggressively, so tokens per GPU-second fall and $/M rises — even on identical hardware.

Same 8×H100 pool, fixed $20/h:
  High batch, ~30 tok/s/user  → 40,000 aggregate tok/s → ~$0.14 / M
  Low batch,  ~80 tok/s/user  → 12,000 aggregate tok/s → ~$0.46 / M

Sell "fast" as a premium SKU, or accept thinner margins. Do not promise both list-cheap tokens and frontier interactivity without the GPU math.

Implications:

• Offer a fast expensive tier and a slower cheap tier. Interactivity and $/token trade on the same GPU. See inference metrics.
• Watch hidden multipliers: retries, tool loops, thinking tokens, and eval traffic in prod.
• Put budgets in the gateway (routing) before marketing scales traffic.

Every technique in the core optimizations group exists to raise tokens/s per GPU or cut GPUs per token: continuous batching, PagedAttention, FlashAttention, quantization, speculative decoding, prefix caching. Stack effects are not always multiplicative. Once you are compute-bound on prefill, more decode tricks help less. Each removed tax should show up in the concurrency sweep.

Common mistakes

• Pricing from mean COGS. p95 users set the margin risk.
• One SKU for cheap tokens and high tok/s/user. Those need different batch points.
• No ceiling on agent loops. 20 model calls per click will eat a $20/mo plan.

Check yourself

1. Same $20/h pool: 40,000 tok/s vs 12,000 tok/s. What happens to $/M? Answer: it rises from about $0.14 to about $0.46.
2. Why price from p95 user cost rather than the mean? Answer: power users, retries and tool loops dominate margin risk.

Key takeaways

• Inference is often the dominant variable cost. Price from p95 COGS.
• Fast tokens/user and cheap $/M fight on the same GPU. Sell them as different tiers.
• Cache-friendly prompts and gateway budgets are product features.

Further learning

Back to the economics of inference.