Long-context techniques

A model trained on short documents does not automatically handle a long prompt. Three things must hold. The positions must extend past the training length. Attention must stay affordable as the length grows. And the model must have learned from long examples how to use distant text. Each fix trades something away.

Stretch the positions: RoPE scaling

Rotary position embeddings (RoPE) are trained for a native length. To go further, change the rotary base or interpolate the positions. Methods such as NTK-aware scaling and YaRN do this.

The change alone is not enough. Continue training on long documents, so the model learns to use the new range. Without that, a model often attends to nearby text and ignores the middle.

Limit the cost: sliding windows and ring attention

Full attention compares every token with every other token, so its cost grows with the square of the length. Two approaches reduce this.

Sliding-window attention lets each token attend only to the previous w tokens. Compare the number of attention pairs for a 128,000-token prompt with a 4,096-token window:

full attention:    128,000 × 128,000 = 16,384,000,000 pairs
sliding window:    128,000 × 4,096   =    524,288,000 pairs
ratio: 16,384,000,000 / 524,288,000 = 31.25

Each layer does about 31 times less attention work. Stacking layers still lets information travel further, because each layer passes information one window forward.

Ring attention splits the sequence across devices. The devices pass key-value blocks around a ring, so no single device holds the full score matrix. This makes very long training contexts possible, but the communication cost is real.

Measure the effective context

A model can find a single needle and still miss a fact in the middle of a long document. Use task suites, not only synthetic retrieval.

Common mistakes

• Claiming a context length from the config alone. Without long-document training, the number is weak evidence.
• Testing only needle retrieval. It misses reasoning over long documents.
• Forgetting the cache. A model that fits a long prompt can still run out of GPU memory during generation.

Check yourself

1. A sliding window of 4,096 tokens covers a 32,768-token prompt. How many times fewer attention pairs does each layer compute than full attention? Answer: About 8 times, because 32,768 / 4,096 = 8.
2. Why does a longer position range alone not guarantee use of the middle of a prompt? Answer: The model must also learn to use distant positions, which takes training on long documents.

Key takeaways

• Long context needs positions, affordable attention and long training data together.
• RoPE scaling extends positions, and continued training teaches their use.
• Sliding windows cut attention work roughly by the ratio of length to window.
• Measure effective context with task suites, not a spec-sheet number.

Further learning

Serving long contexts has its own costs. See long-context serving.

Next, see architectures that avoid quadratic attention altogether: state space models and hybrids.