Total vs active parameters

A mixture-of-experts (MoE) model splits its feed-forward layers into many small networks called experts. For each token, a small router picks a few of them and skips the rest. The model can hold hundreds of billions of parameters while only a few tens of billions run for any one token.

The big idea

Think of a hospital with 200 specialists. Each patient sees only two. The building still needs room for all 200 specialists, so memory is high. But each patient costs only two consultations, so the work per patient is low.

The picture breaks down in one place. In a real MoE, a token's experts can sit on different GPUs. Sending the token there costs time, which the hospital analogy hides. That cost is the subject of the next pages.

Walking through the numbers

DeepSeek-V3 is a useful example. The model report gives about 671B total parameters and about 37B active per token.

Trace the arithmetic, using the rule from inference math that a forward pass costs about 2 × parameters FLOPs per token:

Weights at FP8 (1 byte each):  671B × 1 byte   = about 671 GB
Active FLOPs per token:        2 × 37B         = about 74 GFLOPs
Dense model of 671B, per token: 2 × 671B       = about 1,342 GFLOPs
Ratio:                         1,342 / 74      = about 18×

Memory is set by the 671B. Compute is set by the 37B. Serving has to respect both.

Routing also means each token touches only a small slice of the expert pool. With 256 experts and k = 2, one token uses 2 of 256 experts, or under 1%.

Common mistakes

• Sizing the GPU count from active parameters. The whole expert pool must sit in memory, so the total decides the fit.
• Assuming fewer FLOPs means faster. Decode can still be slow if weights are spread badly or tokens must move between GPUs.
• Treating an MoE like a dense model with the same active size. Its serving cost depends on routing and communication.

Check yourself

1. A model has 300B total and 20B active parameters. Roughly how much memory do its FP16 weights need, and how much compute does one token need? Answer: about 600 GB for weights (2 bytes × 300B), and about the compute of a 20B dense model (about 40 GFLOPs per token).
2. With 64 experts and k = 2, what share of the experts does one token use? Answer: 2 / 64, about 3%.

Key takeaways

• An MoE stores its total parameters but computes with its active parameters.
• Memory fit follows the total. FLOPs per token follow the active count.
• The router picks k experts per token. The rest sit idle for that token.
• The hard part of serving MoEs is moving tokens to the right experts. The next page covers that.

Further learning

Next, read expert parallelism and all-to-all. For the original sparse-routing design, see Switch Transformers.