Data parallelism for inference

Data parallelism (DP) in inference means independent replicas of a model that is already sharded with tensor or pipeline parallelism. Each replica has its own KV cache pool. A load balancer or router sends new requests to a replica.

DP does nothing for single-request latency. If one replica's time per output token is 40 ms, ten replicas still deliver about 40 ms. They deliver ten times the tokens per second of aggregate capacity.

The big idea

Think of ten identical coffee shops. Opening more shops serves more customers. It does not make any one espresso faster.

The picture breaks because KV cache is sticky. A follow-up turn wants the replica that already holds its prefix. Dumb round-robin fights prefix caching.

Walking through a capacity sketch

You need 2,000 decode tokens/s aggregate, at time per output token of 50 ms or less.

One TP=2 replica delivers 400 tokens/s at the concurrency sweet spot
Replicas needed          = ceil(2000 / 400) = 5
Plus about 20% headroom  = 6 replicas
GPUs                     = 6 × 2 = 12

Now compare a wider replica. If TP=4 delivers 550 tokens/s but uses 4 GPUs:

Tokens/s per GPU at TP=2:  400 / 2 = 200
Tokens/s per GPU at TP=4:  550 / 4 = 137.5

Pick the layout with better tokens/s per GPU under the SLO, not the lowest time per token in isolation. Then multiply replicas until goodput meets the traffic plan.

Stateful wrinkle: prefix and KV reuse want sticky or cache-aware routing so a follow-up lands where its prefix lives.

Common mistakes

• Raising TP to raise QPS. TP can cut latency on one replica. QPS comes from more replicas.
• Round-robin across replicas when prefixes matter. Hit rate collapses. See cache-aware routing.
• Ignoring whale sessions. One replica pinned by a 200k-token KV leaves the others idle. Cap max context per replica, or shed long sessions to a long-context pool.

Check yourself

1. You add nine replicas to a replica that runs at 40 ms per token. What happens to one user's stream? Answer: it still runs at about 40 ms per token. Aggregate throughput rises about 10×.
2. Why can one busy session unbalance a DP fleet? Answer: its KV lives on one replica. Other replicas cannot steal that cache, so they sit idle while that replica is pinned.

Key takeaways

• DP replicas scale throughput, not per-request latency.
• Each replica owns its KV pool, so routing must respect prefixes.
• Size the fleet from tokens/s per GPU under the SLO, then add headroom.
• After the model fits with TP or PP, scale QPS with DP.

Further learning

Next, context parallelism shards very long sequences. For the training cousin of this idea, see data parallelism and all-reduce.