Inference arithmetic

Before you choose a serving engine or tune a flag, you need a quick calculator for models and GPUs. Inference arithmetic answers four questions: will the weights fit, how slow is one stream, how many sequences can run at once, and where does batching stop helping.

Wrong answers lead to the wrong GPUs or to latency targets that cannot be met. Right answers give every later choice, such as batching, quantization or parallelism, a solid baseline to adjust.

The numbers here assume H100-class hardware: about 80 GB of HBM, about 3.35 TB/s of memory bandwidth and about 989 TFLOPS of dense BF16 tensor math. Treat them as orders of magnitude and check the datasheet for your SKU.

Why this matters

Serving costs come from two limits: how many bytes must move each step, and how many FLOPs must run. Most mistakes come from using the wrong limit for the question. Napkin math tells you which limit applies before you spend GPU hours on a test.

Concepts in this section

Step	Concept	What it answers
1	Weight memory	Do the weights fit on the GPUs I have?
2	Decode latency floor	How fast can one stream possibly go?
3	Prefill time	How long before the first token appears?
4	Batch ceiling	When does a bigger batch stop helping?

Putting it together

Take Llama-3-70B in FP8 on two 80 GB H100s, with tensor parallelism across both.

Weights fit. 70 GB split over 2 GPUs is 35 GB each.
KV budget. About 160 GB total minus 70 GB of weights minus about 10 GB of overhead leaves about 80 GB for the cache.
Tokens that fit. At about 320 KB per token in BF16, 80 GB holds roughly 250,000 token slots.
Concurrency. One 4,000-token chat needs about 1.28 GB, so 80 GB holds about 60 such chats. Paging and fragmentation make this lower in practice.
Single-stream speed. About 10 ms per token, or about 96 tokens/s.
First token. A 1,000-token prompt takes about 0.12 s to prefill.

The answer to "can we serve 50 concurrent 4k chats on this?" is "tight, but not absurd."

Common mistakes

Using peak FLOPS to predict chat speed. Single-stream decode is limited by memory bandwidth.
Stopping at "the weights fit." Users also need KV cache room.
Comparing throughput across blogs without matching batch size, lengths and precision.
Planning from average context when the SLO is about the long prompts at p99.
Reading MoE parameter counts as compute. Memory follows the total stored weights. FLOPs follow the active ones.

Key takeaways

Weight memory is parameters times bytes per parameter. Precision and parallelism decide whether the model loads at all.
Single-stream decode is limited by memory bandwidth. Prefill is limited by compute.
Batching reuses each weight read until compute or KV memory runs out.
Napkin math sets the feasible region. Benchmarks pick the operating point inside it.