Hot experts and load imbalance

In an MoE, the router decides which experts each token visits. Real routers do not spread tokens evenly. Some experts get far more tokens than average. These are hot experts. Their GPUs run at full load while the others wait, and the combine step makes every GPU wait for the slowest one. One hot expert can slow a whole fleet, even when average GPU utilization looks fine.

The big idea

Picture a shop with eight checkout lanes. One lane gets ten times the shoppers of the others. Average lane use may look moderate, but the shoppers in the long lane are the ones who notice.

The picture breaks down in one way. A slow checkout lane does not hold up the rest of the store. In an MoE, the combine step is a barrier for the whole layer, so one slow expert does hold up every other GPU.

Walking through the imbalance

Take a decode step with 64 tokens, k = 2, and 8 experts on 8 GPUs (one expert each). That is 128 expert assignments.

Fair share per expert     = 128 / 8 = 16 tokens
Expert 3 gets             = 80 tokens       (5× its fair share)
Other 7 experts get       = 48 tokens       (about 7 each)
Imbalance ratio, expert 3 = 80 / 16 = 5×

GPU 3 has five times the work of a typical GPU. The other seven finish early and then wait at the combine. The layer runs at GPU 3's pace.

Mitigations, roughly in the order teams use them:

• Replicate hot experts. Copy them onto extra GPUs so their tokens can go to any copy. This is the idea behind expert placement load balancing (EPLB).
• Re-place experts periodically. Measure tokens per expert and repack the experts onto GPUs. This is costly, so do it rarely.
• Use larger batches. A bigger batch smooths out the noise in routing.
• Cap capacity and drop overflow. Overflow tokens are dropped or rerouted. This hurts quality, so treat it as a last resort.

What to measure

Track these instead:

• Tokens per expert per step, compared with the fair share (the imbalance ratio).
• All-to-all wait time per MoE layer, especially p99.
• Router top-k histogram over a sliding window.
• Idle time waiting on combine, which is the signature of stragglers.

A sustained imbalance ratio of about 2–3× is a signal to rebalance or replicate the hot set.

Common mistakes

• Judging balance by average GPU utilization. Use per-expert token counts and wait times.
• Dropping tokens by default. Capacity limits change outputs. Use them only when nothing else works.
• Testing on one traffic mix. Expert load shifts with the product mix, such as code versus chat versus agents.

Check yourself

1. One expert gets three times its fair share of tokens. Why does the whole layer slow down, not just that GPU? Answer: the combine waits for every expert's output, so the slowest GPU sets the pace.
2. Why does replicating a hot expert help? Answer: its tokens can go to any replica, so one GPU no longer takes all of them.

Key takeaways

• Routers are uneven, so some experts get many more tokens than others.
• The combine step waits for the slowest expert GPU, so one hot expert slows the layer.
• Measure tokens per expert against the fair share, and all-to-all wait time.
• Fix imbalance with replicas and re-placement first. Dropping tokens is the last resort.

Further learning

Next, read different EP widths for prefill and decode. For a short primer on why all-to-all exists and how imbalance hurts it, see this video.