Context parallelism

When n exceeds what tensor-parallel heads can hold, context parallelism (CP) shards the sequence across devices. Ring attention (Liu et al.) passes KV blocks around a ring of GPUs. Each rank computes blockwise attention on its local queries and overlaps communication with compute when blocks are large enough.

The big idea

A long novel is split across desks. Each desk reads its chapter against a rotating stack of notes from the other desks. When the notes are fat (full prefill), the rotation is worth it. When you only ask one question (decode), the rotation is a lot of walking for one sentence.

That is why production guidance is: use CP on prefill pools, keep decode on TP/DP layouts. A natural fit for disaggregated serving.

Walking through it

Inference-oriented variants (Meta's pass-KV / pass-Q):

• Pass-KV: circulate KV. Strong for full prefill and low cache-hit partial prefills. Communication volume suits big K/V blocks.
• Pass-Q: circulate queries. Better when KV is huge and already resident (decode, high hit-rate partial prefill) because Q is smaller per step.

CP cuts long prefill latency near-linearly with enough GPUs and good overlap. On decode, CP often hurts TPOT: extra collectives for little parallel work on a single query row.

Meta quotes 128k prefill ~3.8 s and 1M prefill ~77 s on up to 128 H100s for Llama 3 405B in FP8. Quote efficiency and GPU count with those numbers. A single-GPU fantasy number is not the same result.

Even without CP, chunked prefill matters more as n grows. A monolithic 200k prefill monopolizes a replica for seconds and destroys TPOT for everyone else. Chunks of 1k–4k tokens let decode steps breathe. At extreme n, chunking alone is not enough. You still pay quadratic FLOPs.

Common mistakes

• Turning on CP for decode because it helped prefill. Measure TPOT. It often gets worse.
• Quoting 1M-token prefill time without GPU count. The result was bought with a ring of cards.
• Skipping overlap. If communication does not hide behind compute, CP is just extra latency.

Check yourself

1. Pass-KV or pass-Q for a full cold prefill? Answer: Pass-KV. The K/V blocks are the payload, and every rank needs them.
2. Why split long prefill onto a CP pool and keep decode elsewhere? Answer: CP helps the quadratic prefill. Decode has one query row and extra collectives often slow it down.

Key takeaways

• CP shards the sequence. Ring attention rotates KV (or Q) and computes blockwise.
• Use it to scale long prefill. Be careful on decode.
• Pass-KV for cold prefill. Pass-Q when KV already lives on the ranks.
• Pair with disaggregation so decode stays responsive.

Further learning

Ring Attention with Blockwise Transformers for Near-Infinite ContextTransformers have emerged as the architecture of choice for many state-of-the-art AI models, showcasing exceptional performance across a wide range of AI applications. However, the memory demands imposed by Transforme…arXiv.org Context Parallelism for Scalable Million-Token InferenceWe present context parallelism for long-context large language model inference, which achieves near-linear scaling for long-context prefill latency with up to 128 H100 GPUs across 16 nodes. Particularly, our method ac…arXiv.org

GPU MODE's Ring Attention lecture:

Next, sliding-window attention is the architectural escape from O(n) KV.