Production and economics
On-device and edge inference
Run models on laptops, phones and browsers when latency, privacy or offline matter — with napkin math for bandwidth-bound decode and the right runtime.
Not every token should cross the public internet. A voice agent that round-trips to a distant GPU adds hundreds of milliseconds before the first syllable. A notes app that uploads every keystroke for autocomplete is a privacy non-starter. A plane-mode coding assistant cannot depend on an API. On-device and edge inference is running useful models where the user, or a nearby gateway, already is.
The same inference arithmetic and quantization ideas apply. Start with the bandwidth napkin: tok/s ≈ B / weight_bytes. A 4-bit 8B (~4.5 GB) on a 200 GB/s laptop is about 40 tok/s. If the ceiling cannot meet UX, shrink the model or hybridize. Do not hope for a magic runtime. Unified memory on Apple silicon and phones means the model, KV, OS and tabs share one pool — leave headroom.
Concepts in this section
Read them in order. Each one builds on the one before.
| Step | Concept | What it answers |
|---|---|---|
| 1 | On-device decode ceilings | How fast can this silicon run this model? |
| 2 | Edge runtimes and GGUF | Which stack and quant for this OS? |
| 3 | Small models and hybrid routing | What stays local, and what goes to the cloud? |
| 4 | KV, thermals and downloads | How do you ship without melting the phone? |
Quality versus size
On-device models rarely match frontier APIs on open-ended reasoning. They often match on narrow tasks after distillation or fine-tuning: classify intent, rewrite tone, extract fields, draft a short reply. If the feature needs deep reasoning, keep it hybrid. Document the quality gap so nobody plans a fully offline frontier assistant from a 3B Q4 demo.
Pick llama.cpp/GGUF for cross-platform desktop, MLX or Core ML for Apple-first, ExecuTorch/ONNX for mobile encoders, WebGPU for zero-install previews. Always A/B quality against the cloud teacher on your eval set before promising parity.
For voice agents, shaving 200–500 ms of network often matters more than a slightly smarter cloud model. Incremental token cost can be near zero on silicon the user already paid for — that flips the cost-per-token story for high-volume, low-stakes features.
Common mistakes
- Ignoring the bandwidth ceiling. A runtime will not turn 4.5 GB of weights on 50 GB/s into 80 tok/s.
- Always-resident 7B on a phone. Battery and heat reviews will sink the app.
- Promising frontier quality offline. Specialize and hybridize instead.
Key takeaways
- On-device wins on latency, privacy, offline and marginal cost — not on engineering simplicity.
- Decode speed is mostly bandwidth / model bytes. A 4-bit ~8B (~4.5 GB) on 200 GB/s is roughly ~40 tok/s.
- Match the runtime to the device. Prefer small specialized models plus hybrid cloud routing.
- Budget for thermals, battery and download size or the product fails outside the demo.
Further learning
Clear walkthrough of ggml/GGUF, quantization types, and the llama.cpp decode loop:
Related
On-device decode ceilings
Decode speed on a laptop or phone is roughly bandwidth divided by weight bytes, and client silicon is far below an H100.
Quantization
INT8, FP8, INT4 and FP4. Fewer bits shrink weights, activations and KV. Match the format to memory-bound decode or compute-bound prefill, then eval quality.
Inference arithmetic
Napkin math for any model on any GPU: weight memory, decode and prefill time, batch limits and KV budget.