Serving long contexts

A million-token prompt is a different product from 8k chat. Prefill attention is quadratic. KV and decode reads are linear and still huge. Serving this well means context parallelism, hierarchical KV, windowed or hybrid attention, and an honest check of whether the model can use the needles you stuffed in the haystack.

Systems work (speed, fit) and eval work (quality) are separate gates. Shipping only the first ships a fast liar.

Concepts in this section

Offload and reuse are covered under KV offloading and prefix caching.

Capacity, caching and price

Prefix caching flips multi-turn long chats: turn 1 pays for the big prefill; later turns should hit cached KV for the static corpus and only prefill the delta. Cache-aware routing is mandatory. A random load balancer turns every turn into a cold million-token bill. Context engineering — pinned prefixes, content-addressed blocks — is what makes the hit rate real.

Decode still pays O(n) KV reads per token unless you window. A 150k-token repo snapshot that hits the cache still makes decode expensive.

Each session at 32k tokens, 0.5 bytes/token/layer effective (illustrative FP8 GQA), 80 layers:

32000 × 80 × 0.5 ≈ 1.28 GB KV per session
40 sessions → ~50 GB KV plus weights

One H100 80 GB replica with a 20 GB weight shard has room, barely, after fragmentation. Raise average length to 64k and you OOM or preempt. Long-context pools need admission control: reject or queue when free KV pages fall below a watermark, rather than thrash-killing everyone's TPOT.

API vendors charge premiums or separate SKUs for 128k / 1M contexts because prefill FLOPs explode, KV residency kills batchability, and high-memory GPUs become the only fit. Cache hits on long system prompts are valuable, hence prompt-caching discounts. Cold long prompts are toxic to margin. Self-host the same way: short vs long pools, different parallelism, different internal prices. See economics of inference.

Speculative decoding still works, but the draft model must share the long KV layout or re-prefill cheaply. The win shrinks if draft prefill on a huge context dominates.

Benchmark speed at fixed n (8k, 32k, 128k, 1M), with and without prefix hits, and sweep concurrency. Benchmark quality with needles at varying depths, multi-needle reasoning, and real tasks. Gate features on effective context, not max_position_embeddings.

Common mistakes

• Sizing GPUs for one max-length session instead of many medium-long ones.
• Skipping KV watermarks so short chat shares a replica with a 200k prefill.
• Trusting marketed context without an effective-context curve.

Key takeaways

• Full attention prefill is ~O(n²). KV and decode reads are ~O(n).
• Ring / context parallelism scales long prefill. Use it carefully on decode.
• Windows and hybrids cap KV when the product allows.
• Price and plan long context as a different SKU. Benchmark effective context, not only speed.

Further learning

Ring Attention with Blockwise Transformers for Near-Infinite ContextTransformers have emerged as the architecture of choice for many state-of-the-art AI models, showcasing exceptional performance across a wide range of AI applications. However, the memory demands imposed by Transforme…arXiv.org

Context Parallelism for Scalable Million-Token InferenceWe present context parallelism for long-context large language model inference, which achieves near-linear scaling for long-context prefill latency with up to 128 H100 GPUs across 16 nodes. Particularly, our method ac…arXiv.org

How to Scale Your Model: All About Transformer InferencePerforming inference on a Transformer can be very different from training. Partly this is because inference adds a new factor to consider: latency. In this section, we will go all the way from sampling a single new to…How To Scale Your Model