Core optimizations
Multi-tier prefix cache
Extend a radix or block cache from GPU HBM to host DRAM and then SSD. Larger working sets, slower hits. Worth it for long-lived sessions, not short one-offs.
HiCache-style designs, in SGLang and in research systems, extend the prefix tree across tiers. GPU HBM is the fastest and smallest. Host DRAM is larger and a PCIe reload away. SSD or a remote memory store is largest and slowest. A hit in L1 skips prefill immediately. A hit in L2 or L3 still beats recomputing a huge prefix, if the reload is cheaper than the math.
Concept
A tier is a storage layer with its own latency and capacity. The cache moves cold prefixes down and hot trunks up, the same idea as a CPU cache, applied to KV blocks.
The big idea
Think of a desk, a filing cabinet, and a warehouse. The desk holds today's letters. The cabinet holds this month's. The warehouse holds the archive. You do not warehouse a sticky note.
The picture breaks for short misses. A naive reload of a small prefix can make TTFT worse than recomputing it. Tiering pays off for long-lived sessions and huge shared corpora, not for one-off short prompts.
Walking through the tiers
| Tier | Medium | Latency | Capacity |
|---|---|---|---|
| L1 | GPU HBM | Hit = skip prefill immediately | Smallest |
| L2 | Host DRAM | Reload over PCIe, then skip compute | Larger |
| L3 | SSD / remote memory store | Higher latency; still beats full prefill for huge prefixes | Largest |
Prefetch and write-back policies matter. Interactive SLOs often keep hot trunks in GPU and demote cold sessions downward. A session that returns after an hour can reload from DRAM. A session that returns after a week might reload from SSD, or just prefill again if the prefix is short.
Common mistakes
- Reloading a short miss. If the prefix is small, recompute. PCIe plus setup can lose to a short prefill.
- Filling L1 with cold tails. Evict leaves first, as in RadixAttention. Keep trunks hot.
- Treating L3 as free. SSD hits still add TTFT. Publish that cost, or keep interactive trunks in L1.
Check yourself
- A 3,000-token system prompt missed in HBM but sits in host DRAM. Is a reload usually worth it? Answer: usually yes. Reloading 3,000 tokens over PCIe is cheaper than prefilling them.
- A 40-token header missed in HBM. Should you fetch it from SSD? Answer: no. Recompute. The reload setup can exceed a tiny prefill.
Key takeaways
- L1 is GPU, L2 is DRAM, L3 is SSD or remote memory.
- Tiering extends the working set for long sessions and huge shared prefixes.
- A short miss should recompute, not reload.
- Keep hot trunks in HBM. Demote cold sessions downward.
Further learning
Back to the prefix caching map. Engine docs: vLLM.
Related
Prefix caching
Reusing KV across requests that share prompts. Block hashing, RadixAttention, cache-aware routing, and why cached input tokens are billed cheaply.
RadixAttention
SGLang stores retained KV in a radix tree keyed by token sequences. Shared prefixes are shared paths from the root, with LRU eviction at the leaves.
The KV cache
What keys and values the cache stores, how big it gets per token, and how GQA, MLA, quantization and offloading shrink it.