Mixture of experts

A dense feed-forward layer sends every token through the same weights. A mixture-of-experts (MoE) layer replaces it with many expert networks and a router. The router sends each token to a small number of experts. The model can hold far more parameters than any single token uses.

The big idea

For each token x, the router scores every expert, keeps the top k (often 1 or 2), and returns a weighted sum of their outputs:

gates = softmax(x · W_router)     over all E experts
keep the top-k gates
y = sum over the top-k of gate_i · Expert_i(x)

Two counts matter, and they differ:

• Total parameters include every expert.
• Active parameters are the ones a token actually uses: k experts plus the attention layers.

So the compute per token looks like a smaller dense model, while the capacity to store knowledge looks like a much larger one.

Walking through one routing step

Take 4 experts with top-2 routing. The router gives the token these weights:

experts:  1     2     3     4
gates:    0.5   0.1   0.3   0.1

The top two are expert 1 (0.5) and expert 3 (0.3). Experts 2 and 4 do no work for this token. One common choice is to rescale the two kept weights to sum to 1:

expert 1 weight = 0.5 / 0.8 = 0.625
expert 3 weight = 0.3 / 0.8 = 0.375
y = 0.625 · Expert_1(x) + 0.375 · Expert_3(x)

With 2 of 4 experts active, half the expert compute runs per token. Mixtral 8×7B works the same way with eight experts and top-2 routing. Its active parameters are closer to a roughly 12 to 13 billion parameter dense model than to its full total.

What makes MoE hard

• Load balancing. Left alone, the router sends most tokens to a few favorite experts. Auxiliary losses, or newer balancing schemes, push traffic toward an even spread.
• Expert parallelism. Experts live on different GPUs. Each layer then needs an all-to-all exchange of tokens, which is communication-heavy.
• Capacity limits. Each expert accepts only a set number of tokens per step. Overflow is dropped or rerouted. Too little capacity hurts quality, and too much wastes compute.
• Shared experts. Some designs, such as DeepSeek's, always run a shared expert alongside the routed ones. This stabilizes common computation.

Common mistakes

• Comparing by total parameters alone. A "70B" label says little until you know the active count.
• Ignoring serving. The routing and the all-to-all traffic change how the model runs in production.
• Blaming the architecture for tuning issues. Poor routing, unbalanced load or wrong capacity often look like a bad model.

Check yourself

1. A router picks the top 2 of 8 experts per token. What fraction of expert networks run for each token? Answer: 2 of 8, or one quarter.
2. Why does a MoE model need a load-balancing signal? Answer: Without one, the router sends most tokens to a few experts. The others sit idle, and the devices holding the busy experts become bottlenecks.

Key takeaways

• An MoE layer routes each token to a few experts out of many.
• Total parameters are large, and active parameters stay modest.
• Load balancing, expert parallelism and capacity limits are the main engineering challenges.
• Always read the active parameter count, not only the total.

Further learning

Mixtral of ExpertsWe introduce Mixtral 8x7B, a Sparse Mixture of Experts (SMoE) language model. Mixtral has the same architecture as Mistral 7B, with the difference that each layer is composed of 8 feedforward blocks (i.e. experts). Fo…arXiv.org

DeepSeek-V3 Technical ReportWe present DeepSeek-V3, a strong Mixture-of-Experts (MoE) language model with 671B total parameters with 37B activated for each token. To achieve efficient inference and cost-effective training, DeepSeek-V3 adopts Mul…arXiv.org

Next, look at how models stretch their context window: long-context techniques.