Expert parallelism and all-to-all

Expert parallelism (EP) gives each GPU a subset of the experts. A GPU runs only the experts it owns, so no GPU has to hold the whole pool. The cost is that tokens must travel. Each token's hidden state goes to the GPUs that own its chosen experts, and the results come back.

The big idea

Think of a post office with several sorting centres. Each letter goes to the centre that handles its address, gets processed, and then goes back to the sender. Every letter makes two trips.

The picture breaks down in size. A letter is small. A token here is a full hidden vector, and an MoE layer makes two trips for every token, in every MoE layer. The trips can cost more than the expert math itself.

Walking through one MoE layer

1. Attention and the router run on each GPU. Every token gets a list of (expert, weight) picks.
2. Dispatch: an all-to-all sends each token's hidden state to the GPUs that own its experts.
3. Each GPU runs its local experts on the tokens it received, as one grouped matrix multiply.
4. Combine: a second all-to-all returns each expert output to the token's home GPU. The home GPU takes a weighted sum into the residual stream.

Traffic math

Per token, per MoE layer, the traffic is about 2 × k × H × bytes. The 2 is for dispatch plus combine. Here H is the hidden size and k is the number of experts per token.

H = 7168, FP8 activations (1 byte), k = 8
Per token, per layer:  2 × 8 × 7168 × 1 = 114,688 bytes = 112 KiB
Across 50 MoE layers:  50 × 112 KiB    = 5,600 KiB  (about 5.5 MiB)

At decode, one token per step is small. At prefill with 8,000 tokens, the same formula gives about 45 GB of MoE traffic. Prefill is a throughput problem. Decode is a latency problem.

How EP mixes with TP and DP

A common layout for large MoEs:

• Attention: modest tensor parallelism (for example TP=4), plus data parallel replicas across the node.
• Experts: expert parallelism across GPUs and nodes, with TP=1 on each expert to avoid stacking more collectives.
• Shared experts and early dense layers: TP=1 or small TP, because every token uses them.

Common mistakes

• TP-sharding every expert. You pay dense-style collectives and still hold all the parameters.
• EP-sharding shared experts. Shared experts run for every token, so they belong on TP.
• Ignoring that combine is a barrier. The slowest expert GPU sets the pace for the layer. See hot experts.

Check yourself

1. Why does the layer need a second all-to-all after the experts run? Answer: the expert outputs live on the GPUs that ran them, but each token's residual stream lives on its home GPU, so the outputs must go back.
2. If the hidden size doubles and everything else stays the same, what happens to MoE traffic? Answer: it doubles, because traffic is proportional to H.

Key takeaways

• EP shards experts across GPUs. Each MoE layer costs two all-to-all steps.
• Traffic per token per layer is about 2 × k × H × bytes.
• Keep attention on TP and DP. Put experts on EP.
• Shared experts stay on TP, not EP.

Further learning

Next, read hot experts and load imbalance. For the communication library used in large deployments, see DeepEP.