Guide

Anatomy of LLM inference

KV offloading

Hot tokens stay in HBM. Warm prefixes spill to CPU or SSD. Hits beat recomputing prefill. Cold million-token decode still pays the bandwidth.

37 of 138 · 2 min read

When context is huge, or you want to reuse KV across requests, keeping everything in HBM forever is impossible. Systems such as LMCache and Mooncake, and provider prompt caches, tier the cache.

Concept

Offloading moves cold KV off the GPU to CPU DRAM, NVMe or a remote store, and brings it back on a hit. Prefix caching is the serving pattern that makes reuse pay: a shared system prompt or document stays resident so the next request skips that prefill. See prefix caching.

The big idea

A desk holds the page you are writing. A filing cabinet holds last week's pages. A warehouse holds last year's. You can store more. You cannot read the warehouse at desk speed.

The analogy breaks on decode of a cold 1M-token context. Offloading helps capacity and TTFT on cache hits. It does not make a cold long decode cheap. You still pay bandwidth when those bytes are read.

Walking through it

Hierarchy:

  1. GPU HBM — hot recent windows and active pages (PagedAttention).
  2. CPU pinned memory — warm tiers with PCIe or NVLink-C2C transfer.
  3. Local SSD or networked KV — multi-user prefix reuse, disaggregated prefill output.

For agent fleets with repeated tool schemas and policies, prefix reuse is often the largest win. Details for million-token prompts live under long-context serving.

Offloading boosts multi-tenant packing. It rarely wins latency races for true million-token interactive decode unless the model is sparse over memory. Prefetch along the attention schedule. Stall if the working set thrashes.

Retrieval-style sparse attention (full recent window plus retrieved blocks) turns the problem into RAG-shaped IO. Quality then depends on retrieval recall, not only bandwidth.

Common mistakes

  • Expecting offload to make cold long decode fast. Capacity yes. Interactive TPOT no.
  • Evicting "unimportant" tokens without evals. That is a quality bug.
  • Ignoring preemption storms. The scheduler evicts sequences to free blocks. Victims repay with recompute or reload, and TTFT tanks.

Check yourself

  1. A shared 20k-token system prompt hits the CPU tier on turn 2. What did you save? Answer: Prefill of those 20k tokens. Decode still reads whatever KV is brought back to HBM.
  2. Why is a cold 1M-token context still expensive after you add SSD offload? Answer: Those bytes must be read to attend. Offload stores them. It does not shrink the working set.

Key takeaways

  • Tier KV: HBM hot, CPU warm, SSD or remote cold.
  • Hits beat recomputing prefill. That is the TTFT win.
  • Offload extends capacity. It does not erase O(n) decode reads.
  • Pair it with prefix caching and cache-aware routing, or every turn is cold.

Further learning

How KV cache and paging interact with GPU memory in practice:

How KV Cache Speeds Up LLMs for Faster AI Models on GPUsIBM Technology

Back to the KV cache hub.

The KV cache

What keys and values the cache stores, how big it gets per token, and how GQA, MLA, quantization and offloading shrink it.

Prefix caching

Reusing KV across requests that share prompts. Block hashing, RadixAttention, cache-aware routing, and why cached input tokens are billed cheaply.

Serving long contexts

Million-token prompts: quadratic prefill attention, linear KV growth, context parallelism, offloading, sparse/hybrid architectures, and how providers price the pain.