Edge KV, thermals and downloads

A model that fits in RAM at launch can still fail ten minutes later. The KV cache grows with every token. Phones throttle. Users notice battery and a 4.5 GB install. Shipping is the difference between a demo and a product.

The big idea

A backpack that fits the tent may not fit the tent plus food plus water as the trip goes on. Weights are the tent. KV is the food. Heat and battery are the hike.

Walking through the budgets

On a 16 GB unified-memory laptop with a 4.5 GB weight footprint, you might only have 4–6 GB left for KV and the OS after browser tabs. A long multi-turn session can OOM or force context truncation long before the model "feels" large.

Rough KV bytes ≈ 2 × layers × num_kv_heads × head_dim × seq × bytes_per_elem. Halving sequence length is often the only viable optimization on a phone — smarter than chasing another 0.5 bit of weight quant.

Practical mitigations:

• Cap context (2k–8k) aggressively for mobile.
• Quantize KV when the runtime supports it.
• Summarize older turns instead of retaining full history.
• Offload cold layers to SSD only if the UX can tolerate hitching (desktop). Phones hate this.

Thermals. Sustained generation heats phones and thin laptops; clocks drop. Design for short generations, a streaming UI that stays useful at 10 tok/s, and pause-on-background.

Battery. High GPU/NPU duty cycles drain fast. Prefer NPU paths when quality allows. Do not keep a large model resident if the feature is rare — load and unload strategically.

App size. A 4.5 GB model inside a mobile IPA/APK is a non-starter for many stores and users. Use on-demand downloads, delta updates and clear storage settings. Desktop can tolerate larger caches.

Checklist:

1. Pick max model size from RAM_headroom / 1.3 (weights + KV + OS fudge).
2. Run the bandwidth napkin. If the ceiling is below the UX target, shrink or hybridize.
3. Choose runtime by OS matrix. Lock quant type in CI with a tiny eval set.
4. Gate downloads behind Wi-Fi and user consent. Show storage used.
5. Measure tok/s on battery + warm device, not only on a cooled desk demo.

Common mistakes

• Ignoring KV growth. The session OOMs after the demo-length chat.
• Measuring only on a plugged-in flagship. Battery and heat are the real device.
• Bundling 4.5 GB in the store binary. Use an on-demand download.

Check yourself

1. 16 GB laptop, 4.5 GB weights, browser tabs open. What is the KV risk? Answer: only a few GB remain, so a long chat can OOM or truncate.
2. Why measure tok/s on battery? Answer: thermals and power limits cut the clock after tens of seconds.

Key takeaways

• Cap context. KV growth, not weight quant, is often the phone constraint.
• Design for heat, battery and pause-on-background.
• Download models on demand, with consent and a storage display.

Further learning

Back to on-device and edge inference.