LLM capacity math

Capacity math for an LLM product starts with users, then turns into tokens, then into GPUs. The numbers are rough on purpose. Interviewers want a clear chain of assumptions, not false precision. Show the formula, state the assumptions, and say how you would calibrate them with real traffic.

The big idea

Work from the daily total to the peak per second, then to the number of GPUs. Each step can use a simple multiplier:

• Daily tokens = daily users × requests per user × tokens per request.
• Average tokens per second = daily tokens ÷ 86,400.
• Peak tokens per second = average × a peak factor, often 2 to 3.
• GPUs = peak tokens per second ÷ tokens per second per GPU, with headroom added.

Cost follows the same chain. A price per million tokens times the daily token count gives the daily bill.

Walking through it

Assume 10 million daily users, each sending 6 requests a day, with about 1,000 tokens per request in and out together. That is 60 billion tokens a day.

daily tokens    = 10,000,000 users x 6 requests x 1,000 tokens = 60,000,000,000
average per sec = 60,000,000,000 / 86,400                      = about 694,000 tokens/s
peak per sec    = 694,000 x 2.5 peak factor                    = about 1,735,000 tokens/s

If one GPU serves 1,000 tokens per second with batching, the peak needs about 1,735 GPUs. Add 20% headroom for failures and deploys, and you have roughly 2,100. Those are rough numbers, but they show the scale: serving costs dominate the budget, and every 10% of efficiency shows up in the bill.

Cost follows the same numbers. At 60 billion tokens a day, each $1 per million tokens costs about $60,000 a day. If the blended price is $2 per million, the daily bill is about $120,000. Routing cheap requests to smaller models is usually the largest saving available.

Be explicit about what the arithmetic leaves out. Retries, tool calls, retrieval queries and long contexts all add tokens. Padding for them is a good habit.

Common mistakes

• Sizing for the average. Peak load sets the fleet. Use a peak factor.
• Forgetting input tokens. Long contexts can dominate output, so count both.
• Treating the GPU count as fixed. Throughput depends on model size, batching and prompt length. Measure it.
• No headroom. A fleet at 100% utilization has no room for a failed node or a deploy.

Check yourself

1. 60 billion tokens a day spread over 86,400 seconds gives what average rate?

   About 694,000 tokens per second, computed as 60,000,000,000 ÷ 86,400.

2. A price of $1 per million tokens and 60 billion tokens a day gives what daily cost?

   About $60,000 a day.

Key takeaways

• Work from users to tokens per day, then to peak tokens per second, then to GPUs.
• Use a peak factor, and add headroom on top.
• Token cost scales the same way as token load, so efficiency gains pay twice.
• State your assumptions, and say how you would measure them.

Further learning

Next: conversation context assembly, which decides how many of those tokens each request uses.