Guide

Scaling out

Why long context is expensive

Prefill attention grows with n squared. KV and decode reads grow with n. A million-token prompt is not eight times a 128k prompt.

89 of 138 · 2 min read

"Just raise max_model_len" is how long-context demos start and how production outages begin. A million-token prompt is not eight times a 128k prompt in wall-clock or cost. Dense prefill attention is quadratic in sequence length. The KV cache is linear in length and can exhaust HBM before FLOPs do. Decode must re-read that cache every token.

Concept

Context length is the maximum token index the model and runtime accept. Effective context is how far back the model still retrieves and reasons reliably, often much shorter than the marketed maximum. Speed and quality are separate gates.

The big idea

A short chat is a hallway conversation. A million-token prompt is a warehouse inventory. Walking the hallway is cheap. Walking every aisle for every new sentence is not.

The analogy breaks on architecture. Sliding windows and hybrids cap the walk. Full attention plus more GPUs does not.

Walking through it

Dense self-attention FLOPs for length n scale about O(n² · d) per layer, plus O(n · d²) for projections and the FFN. For short prompts the FFN dominates. At 100k–1M tokens attention dominates unless you use sparse or linearized variants.

Suppose at n = 8k attention is 25% of prefill time and FFN plus projections are 75%. Moving to n = 128k (16×) multiplies attention work about 256× while linear terms grow 16×. Attention becomes the wall.

FlashAttention reduces memory traffic and lets large n run without materializing n × n matrices. It does not change the asymptotic FLOP count of full attention.

Meta's context-parallelism report (Llama 3 405B, FP8) quotes order-of-magnitude anchors: 128k prefill ~3.8 s and 1M prefill ~77 s on up to 128 H100s with high parallel efficiency. Those numbers exist because they scale GPUs with n, not because one GPU got magic.

KV bytes stay linear, and the constant is brutal. n = 1_000_000, L = 80, GQA with h_kv · d_h = 1024, FP16:

2 · 1e6 · 80 · 1024 · 2 ≈ 327 GB

That is multiple H100s of HBM just for cache, before weights. MLA, FP8 or INT4 KV, fewer stored layers, or windowing is mandatory at this extreme.

Decode TPOT suffers too. Each new token attends to all n past positions, so memory traffic per decode step grows with n. Flash-decoding and paged KV help utilization. They do not remove the O(n) read.

Common mistakes

  • Scaling cost as if it were linear in n. Prefill attention is quadratic for dense full attention.
  • Crediting FlashAttention with cheaper FLOPs. It fixes IO, not the O(n²) math.
  • Sizing the fleet for one max-length session. Reality is many medium-long sessions.

Check yourself

  1. Length grows 16× from 8k to 128k. About how much does dense attention work grow? Answer: About 256×, because FLOPs scale with n².
  2. How much FP16 KV is 1M tokens at 80 layers and h_kv · d_h = 1024? Answer: About 327 GB, before weights.

Key takeaways

  • Full-attention prefill is about O(n²). KV and decode reads are about O(n).
  • Both dominate at 100k+.
  • FlashAttention fixes IO, not FLOP asymptotics.
  • Million-token full attention is a multi-GPU problem, not a flag.

Further learning

Next, context parallelism is how you split n across GPUs.

Serving long contexts

Million-token prompts: quadratic prefill attention, linear KV growth, context parallelism, offloading, sparse/hybrid architectures, and how providers price the pain.

The KV cache

What keys and values the cache stores, how big it gets per token, and how GQA, MLA, quantization and offloading shrink it.

FlashAttention

Why standard attention is HBM-bound, how tiling plus online softmax computes exact attention in on-chip SRAM, and what FlashAttention-2/3 and decode variants change.