Inference as COGS

If the product calls a model on every user action, those tokens are COGS. Classic SaaS hosting is cheap and similar across customers. AI products have margin variance: a median user can look like 90% gross margin while a power user loses money on the same plan.

Walking through a $20 consumer plan

Assumptions (illustrative mid-2020s API pricing — re-quote before you ship):

• Blended inference ≈ $0.005 per request after caching and routing
• Median user: 200 requests / month → COGS = 200 × 0.005 = $1.00
• Power user (top 5%): 4,000 requests / month → COGS = 4,000 × 0.005 = $20.00
• Payment fees ≈ 3% of revenue ≈ $0.60

Median user: $20 − $1 − $0.60 = $18.40 → margin 92%

Power user: $20 − $20 − $0.60 = −$0.60 → margin −3%

If the book is 95% median-like and 5% power users:

Average inference = 0.95 × $1 + 0.05 × $20 = $1.95
Average gross profit = 20 − 1.95 − 0.60 = $17.45  → GM ≈ 87%

That looks healthy — until power users become 20% of the base (common once the product is useful for work):

Average inference = 0.80 × $1 + 0.20 × $20 = $4.80
Gross profit = 20 − 4.80 − 0.60 = $14.60  → GM ≈ 73%

Push power-user share to 40%, or double token prices on the large model, and you land in the 50–60% band that scares boards — or worse if you do not rate-limit.

Compare to classic SaaS on the same $20 plan with $2 hosting + $0.60 fees → GM = (20 − 2.60) / 20 = 87%, and almost no user-level variance.

Team plan with seats

B2B AI tool: $50 / seat / month, 10 seats → $500 / month.

Inference $8/seat → $80. Hosting / vector DB $40. Support $30. Fees 3% → $15.

COGS = 80 + 40 + 30 + 15 = $165
GM   = (500 − 165) / 500 = 67%

If three seats are analysts at $40 of tokens and seven burn $4:

Inference = 3×40 + 7×4 = $148
COGS      ≈ 148 + 40 + 30 + 15 = $233
GM        = (500 − 233) / 500 = 53.4%

Same contract, different mix. That is why AI companies instrument COGS per tenant and put fair-use caps in the MSA.

Common mistakes

• Trusting the average. The average looks fine while the top slice is unprofitable.
• Flat $20 with uncapped frontier-model usage. That is a charity, not a plan.
• Skipping per-tenant cost. You cannot price or cap what you do not measure.

Check yourself

1. Median GM is 92% and 5% of users cost $20 on a $20 plan. Is the book safe? Only while power users stay rare. At 20% share, blended GM is already ~73%.
2. Why put fair-use caps in the MSA? Same seat price, different token mix — three heavy seats can drop a 10-seat deal from 67% to 53% GM.

Key takeaways

• Inference tokens are COGS. Book them there.
• Power users dominate the cost curve. Averages hide them.
• Classic SaaS has little user-level margin variance. AI products do.
• Instrument COGS per tenant before you offer unlimited plans.

Further learning

Levers that move the number: gross margin levers.

The New Business of AI (a16z)At a technical level, artificial intelligence seems to be the future of software. AI is showing remarkable progress on a range of difficult computer science problems, and the job of software developers – who now…Andreessen Horowitz