Weight memory

A model's weights are the learned numbers that define it. Serving starts by loading them into GPU memory, so their size is the first thing to check. The rule is simple: weight bytes equal the number of parameters times the bytes each parameter takes.

The big idea

Think of each parameter as a cell that needs room for its number. The width of the number depends on precision. Halve the width and you halve the room.

The analogy breaks for training. Training also stores gradients and optimizer states. Serving only needs the weights, plus the KV cache covered later.

Walking through it

weight_bytes ≈ params × bytes_per_param

Now check an 80 GB H100 against two models:

• 8B in BF16: 8e9 × 2 = 16 GB. Fits with plenty of room.
• 70B in BF16: 70e9 × 2 = 140 GB. Does not fit on one GPU. You need two GPUs or a lower precision.
• 70B in FP8: 70e9 × 1 = 70 GB. Fits in theory, but leaves little room for the KV cache. Production usually still uses two or more GPUs.

Common mistakes

• Treating "fits in memory" as "can serve users." Weights are the fixed cost. The cache for live requests comes on top.
• Forgetting the bytes per parameter. INT4 is half a byte, not four bits of overhead.
• Using training rules. Gradients and optimizer states do not matter for inference.
• Mixing up MoE sizes. A mixture-of-experts model stores all experts in memory, even though each token uses only some of them. See serving MoE models.

Check yourself

1. How much memory do 13 billion parameters take in FP16? Answer: 13e9 × 2 bytes = 26 GB.
2. Why does FP8 make a 70B model easier to fit than BF16? Answer: It halves the bytes per parameter, from 140 GB to 70 GB, which still leaves little room on one 80 GB card.

Key takeaways

• Weight bytes = parameters × bytes per parameter.
• Precision is the biggest lever. BF16 to FP8 halves the footprint.
• Fitting the weights is necessary but not enough. The cache must fit too.
• Next, check how fast a single stream can go in the decode latency floor.

Further learning

The roofline model explains why these byte counts, and not FLOPs, set decode speed.