Sliding-window and hybrid attention

Full attention plus context parallelism is what you reach for when the product truly needs lossless long-range attention and you can pay GPUs. Many products do not. Sliding-window and hybrid stacks cap KV growth in the model itself.

The big idea

A security camera that only keeps the last ten minutes is a sliding window. One that also keeps a still from every hour is a hybrid. You will miss a face from eleven minutes ago unless a global layer or a sink token kept it.

The picture breaks on tasks that need every token, lossless, at once. Then you pay for full attention.

Walking through it

For serving, windows and hybrids are gifts: predictable memory, simpler than million-token full attention.

Even cache hits leave decode expensive at 150k tokens if every layer is dense. A code assistant with a 150k-token repo in context still pays O(n) KV reads per token unless you window. That is why hybrids show up in long-context model design, not only in serving tricks.

Positional schemes are a serving gotcha. RoPE variants (NTK-aware, YaRN, LongRoPE) stretch or interpolate frequencies so checkpoints trained at 8k can run at 128k with fine-tuning. Load the exact rope scaling config the checkpoint expects. A silent mismatch destroys long-range attention. Cap max_position_embeddings to what you eval'd, even if the GPU could allocate more KV. Naive left-truncation in the API is a quality bug dressed as a 200 OK.

Multi-modal long context (hour-long video plus text) multiplies tokens via patch or frame embeddings. Token accounting in the rate limiter should use real model tokens, including vision.

Common mistakes

• Raising max_model_len on a windowed model and expecting full-document reasoning. The window did not grow.
• Mismatched RoPE scaling at serve time. Long-range attention silently dies.
• Paying CP-and-full-attention prices for a product that would accept a hybrid.

Check yourself

1. A layer has a 4k sliding window. How does its KV grow from token 4,001 to 100,000? Answer: It does not, past the window. Only the latest 4k tokens stay in that layer's cache.
2. Why keep max_position_embeddings at the eval'd length? Answer: The GPU can allocate more KV than the model can use. Extra length is a quality bug, not headroom.

Key takeaways

• Windows cap KV per local layer. Hybrids add a cheap global path.
• Prefer them when the product allows. Full attention plus CP when it does not.
• Serve the RoPE config the checkpoint was trained with.
• Windowed decode is why long cache hits can still be fast enough to ship.

Further learning

See modern architectures for how these layers are trained. Back to serving long contexts for capacity and pricing.