Guide

Scaling out

Different EP widths for prefill and decode

Prefill and decode send very different numbers of tokens to each expert, so the best expert-parallel width is usually different for each phase.

83 of 138 · 2 min read

Expert-parallel (EP) width is how many GPUs share the expert pool. Prefill and decode send very different numbers of tokens to each expert. So one EP width is rarely the best for both. Prefill is rich in tokens per expert. Decode has very few.

Concept

In prefill, a request's whole prompt is processed in one go, so experts see many tokens at once. In decode, each sequence adds one token per step, so each expert sees only a handful. For the full split, see prefill vs decode.

The big idea

Think of a call centre. In a busy hour, each agent has many calls to take, so a few agents on one floor is efficient. At night, a few calls come in, and spreading them across many floors keeps any one agent from sitting on a backlog while others sleep. The analogy works for the idea. It breaks because the floors are GPUs, and moving calls between them costs real network time.

Walking through the decode case

Take a decode batch of 256 tokens with k = 2 and 256 experts. That is 512 assignments, or about 2 tokens per expert on average.

EP = 64 (4 experts per GPU):
  each GPU sees about 512 / 64 = 8 expert-tokens per step
  small, but workable with grouped kernels

EP = 8 on one node (32 experts per GPU):
  each GPU holds 32 experts, many empty in a given step
  a few hot experts on a GPU serialize behind each other

Wider EP helps because each GPU owns fewer experts, so a hot expert is not stacked with 31 others on the same GPU. The cost is more traffic across nodes. Wider EP helps only if placement stays balanced.

Prefill and decode want different things

PrefillDecode
Tokens per expertManyFew
Good EP widthModerate, inside one NVLink islandWide, across more GPUs or nodes
All-to-all styleHigh throughput, bulk transfersLow latency, RDMA, little GPU time taken from compute
Main metricTime to first tokenTime per output token

Prefill needs its big matrix multiplies to overlap with communication, and it wants a compact group. Decode needs its tiny batches delivered fast, and it wants experts spread out.

Trade-off

A wider EP group lowers hot-spot serialization but adds cross-node traffic. A narrower group keeps traffic on fast local links but leaves more work per GPU. Pick the width per phase.

Common mistakes

  • Using one EP setting for both phases. Tune prefill and decode separately.
  • Benchmarking only one phase. A setting that looks good for prefill can hurt decode latency.
  • Widening EP without balancing placement. More GPUs only help if the hot experts are spread out. See hot experts.

Check yourself

  1. Why does a wider EP group often help decode more than prefill? Answer: decode has few tokens per expert, so spreading experts out keeps a hot expert from sharing its GPU with many others.
  2. Why not use the same EP width everywhere? Answer: prefill wants compact, high-throughput transfers. Decode wants wide spread and low latency.

Key takeaways

  • Prefill has many tokens per expert. Decode has very few.
  • Prefill usually prefers a compact EP group. Decode usually prefers a wide one.
  • Wider EP reduces hot-spot serialization only if placement is balanced.
  • This split is why MoE fleets often run prefill and decode in separate pools.

Further learning

Next, read the hub on serving mixture-of-experts models for the whole picture. For the communication library that supports separate prefill and decode modes, see DeepEP.

Serving mixture-of-experts models

Expert parallelism, all-to-all dispatch/combine, and load balancing for sparse MoEs like DeepSeek-V3 — where memory tracks total params but FLOPs track active ones.

Disaggregated prefill and decode

Running prefill and decode on separate GPU pools so each phase gets its own parallelism, batching, and hardware — and when colocated chunked prefill is enough.

Prefill vs decode

The two phases of LLM inference. Prefill is compute-bound and sets TTFT. Decode is memory-bound and sets TPOT. They interfere when they share a GPU.