Serving mixture-of-experts models

A dense 70B model uses all 70B parameters for every token. A mixture-of-experts (MoE) model can hold hundreds of billions of parameters and run only a few tens of billions per token. Serving one well is a different problem from serving a dense model. The bottleneck moves from an all-reduce in every layer to an all-to-all in every MoE layer. It also moves from even matrix multiplies to hot experts, where one overloaded GPU holds up the rest.

The core rule: memory follows total parameters. Compute, and most decode weight reads, follow active parameters. Keep that split in mind and the rest follows.

Before you start, read parallelism for inference and prefill vs decode.

Concepts in this section

Mixing EP with TP and DP

Do not shard an MoE the way you shard a dense model. A layout that works well for large MoEs:

• Attention: modest tensor parallelism plus data parallel replicas.
• Routed experts: expert parallelism across GPUs, with TP=1 on each expert.
• Shared experts and early dense layers: TP=1 or small TP, because every token uses them.

Quantization and MoE

Lower-precision expert weights, such as FP8 or INT4, shrink the total footprint. That lets more experts fit on each GPU and can narrow the EP width. Keep the router and shared experts at higher precision. Aggressive quantization there shifts top-k choices and makes imbalance worse.

MoE does not shrink the KV cache by itself. A large MoE can still be KV-bound on long chats even when its experts fit comfortably. See quantization and the KV cache.

Large-scale EP

DeepSeek-V3's technical report and the DeepEP library describe the pattern used at large scale:

• Prefer EP over very wide TP for MoE layers.
• Use two all-to-all modes. High-throughput kernels for prefill, which moves bulk tokens. Low-latency RDMA kernels for decode, which moves few tokens and must not steal much GPU time from compute.
• Route hierarchically: InfiniBand between nodes, NVLink within a node.

You do not need DeepSeek's topology to serve Mixtral-class models on 2 to 8 GPUs. Once the expert pool is bigger than one node, you are in the same design space: custom all-to-all and separate tuning for each phase.

Common mistakes

• Sizing the cluster from active parameters. The whole expert pool has to fit in memory.
• Treating an MoE as a dense model. Its speed depends on routing and communication.
• Benchmarking on one traffic mix. Expert load changes with code, chat and agent traffic. Test on yours.
• Using one EP setting for prefill and decode. Tune each phase on its own.

Key takeaways

• Memory follows total parameters. Compute follows active parameters.
• Expert parallelism costs two all-to-all steps per MoE layer.
• Hot experts waste a fleet. Use replicas, re-placement and per-expert metrics.
• Prefill and decode want different EP widths and communication kernels.
• Keep attention on TP and DP, and experts on EP. Do not TP the whole MoE.

In practice

When bringing up an MoE in vLLM, SGLang or TensorRT-LLM:

1. Fit the total weights with EP, and check that the expert count divides evenly across GPUs.
2. Benchmark prefill and decode separately. MoE decode usually needs higher concurrency to look good.
3. Enable the all-to-all backend that suits each phase, if your engine offers one.
4. Watch expert histograms under your own traffic.
5. Compare tokens per second per GPU at the same latency target against a dense model with similar active FLOPs. If the MoE loses, check the batch and the EP setup first.

Further learning

For the design space in one talk, see vLLM office hours on scaling MoE. For a short primer on expert-parallel all-to-all, see this video. Papers: DeepSeek-V3 Technical Report and Switch Transformers.