Production and economics
On-device decode ceilings
Decode speed on a laptop or phone is roughly bandwidth divided by weight bytes, and client silicon is far below an H100.
A laptop or phone can run a useful model, but decode speed is set by memory bandwidth, not by how clever the runtime looks. The same inference arithmetic applies. The constants get uglier: hundreds of GB/s instead of ~3 TB/s, and RAM shared with the OS.
Concept
Memory-bandwidth-bound decode. Generating the next token mostly streams weights (and KV) from memory. If the model footprint is W bytes and bandwidth is B bytes/s, a rough ceiling is tok/s ≈ B / W. Faster ALUs do not help until you shrink W or raise B. See the roofline.
The big idea
A truck that must haul the whole warehouse past the dock for every item is decode. Narrow the warehouse (quantize) or widen the dock (faster memory). Client silicon is a narrow dock.
Four reasons dominate the product choice: latency (no WAN RTT), privacy (weights stay on the device), offline, and marginal cost near zero once the user's silicon is paid for. Edge is not automatically cheaper for you as a vendor. You still pay for distillation, app size, OEM testing and a thousand SKUs.
Walking through the napkin
Server GPUs win on HBM (H100-class on the order of ~3 TB/s). Client silicon is different:
| Class | Memory model | Ballpark bandwidth | Implication |
|---|---|---|---|
| Apple silicon (M-series) | Unified memory CPU/GPU/NPU | often ~100–800 GB/s | Strong local LLMs; RAM shared with OS/apps |
| Modern phone SoC | Limited DRAM + NPU | far below laptop; thermals throttle | Prefer 1B–3B or short generations |
| PC dGPU (consumer) | Dedicated VRAM | hundreds of GB/s to ~1 TB/s | Closest to a mini server |
| Browser (WebGPU) | Whatever the OS exposes | highly variable | Fine for demos; worst-case skew is huge |
Unified memory means the model, KV, OS and browser tabs fight for the same pool. A "64 GB Mac" does not give you 64 GB for weights.
Take an 8B model quantized to roughly 4 bits per weight.
Weight bytes ≈ 8e9 × 0.5 bytes ≈ 4.0 GB
Plus norms, embeddings, runtime ≈ 4.5 GB working setOn a laptop with 200 GB/s effective bandwidth:
tok/s ceiling ≈ 200 / 4.5 ≈ 44 tok/sCall it ~40 tok/s after KV reads, sampling and non-ideal kernels. That is the "feels interactive" band for chat. Halve the bandwidth (thermal throttle, competing apps) and you are at ~20 tok/s. Double the model to 70B at 4-bit (~35–40 GB) on the same bus and the ceiling collapses to single-digit tok/s.
Phone with ~50 GB/s effective and a 3B 4-bit model (~1.7 GB weights): ceiling ≈ 50 / 1.7 ≈ 29 tok/s before thermals. After 30 seconds of generation, many phones throttle. Design UX for short bursts.
Prefill of a long prompt can be compute-bound on NPUs with enough FLOPs. Edge products that paste a 20-page PDF into the prompt hit that wall hard.
Common mistakes
- Hoping a runtime beats the bandwidth ceiling.
tok/s ≈ B / Wis physics, not a library bug. - Giving the model the whole RAM number. OS, KV and tabs need headroom.
- Planning novel-length sessions on a phone. Thermals will cut the clock.
Check yourself
- 4.5 GB weights, 200 GB/s. About what is the decode ceiling? Answer: 200 / 4.5 ≈ 44 tok/s, about 40 after overhead.
- Why does a 64 GB Mac not give you 64 GB for weights? Answer: unified memory is shared with the OS, apps and the KV cache.
Key takeaways
- On-device wins on latency, privacy, offline and marginal cost.
- Decode ceiling ≈ bandwidth / weight bytes. 4-bit 8B on 200 GB/s ≈ 40 tok/s.
- Phones want 1B–3B and short bursts. Prefill of huge pastes is a separate wall.
Further learning
Next, pick a stack in edge runtimes.
Related
On-device and edge inference
Run models on laptops, phones and browsers when latency, privacy or offline matter — with napkin math for bandwidth-bound decode and the right runtime.
The roofline model and arithmetic intensity
Predicting performance from FLOPs and bytes: compute-bound vs memory-bound. Start here for the map, then work the examples.
Quantization
INT8, FP8, INT4 and FP4. Fewer bits shrink weights, activations and KV. Match the format to memory-bound decode or compute-bound prefill, then eval quality.