Guide

Production and economics

Small models and hybrid routing

Run a tiny on-device model for easy or private work, and escalate hard queries to the cloud with consent.

128 of 138 · 2 min read

Edge quality usually comes from smaller specialized models, not from cramming a frontier MoE onto a phone. Distill or fine-tune 1B–8B models for the exact task (rewrite, classify, extract, speech). Escalate the hard cases.

On-device models rarely match frontier APIs on open-ended reasoning. They often match on narrow tasks after distillation or fine-tuning. Define success as task metrics on your distribution, not MMLU envy.

The big idea

A receptionist who handles the routine and calls a specialist is faster and more private than putting the specialist on every call. The analogy breaks if you pretend the receptionist is the specialist. Document the quality gap so nobody plans a "fully offline ChatGPT" from a 3B Q4 demo.

Walking through a hybrid path

  • Use retrieval on-device sparingly. Huge local corpora fight the same RAM budget as weights.
  • Run a tiny on-device model for autocomplete, PII scrub or intent. Escalate hard queries to cloud with user consent.
  • If the feature needs deep reasoning, keep it hybrid: local draft plus cloud verify, or cloud-only with a privacy mode that disables the feature.

The product feels fast and private by default without pretending a 3B model is a researcher.

Common mistakes

  • Forcing a frontier-sized model onto the edge. The bandwidth ceiling will not move.
  • Planning offline ChatGPT from a 3B demo. Narrow-task evals, not leaderboard envy.
  • On-device RAG of a huge corpus. The corpus steals the RAM the weights need.

Check yourself

  1. A phone feature classifies intent and sometimes needs a long reasoned answer. What runs where? Answer: intent on-device; the reasoned answer in the cloud, with consent.
  2. Why is a huge on-device retrieval corpus a problem? Answer: it fights the same RAM budget as the weights and the KV cache.

Key takeaways

  • Specialize 1B–8B models for the task. Do not cram a frontier model onto a phone.
  • Hybrid routing: local for easy or private, cloud for hard, with consent.
  • Judge quality on your task metrics, and write the gap down.

Further learning

Next, ship it without melting the phone in edge KV, thermals and downloads.

On-device and edge inference

Run models on laptops, phones and browsers when latency, privacy or offline matter — with napkin math for bandwidth-bound decode and the right runtime.

On-device decode ceilings

Decode speed on a laptop or phone is roughly bandwidth divided by weight bytes, and client silicon is far below an H100.

Edge runtimes and GGUF

llama.cpp and GGUF for cross-platform desktop, MLX or Core ML for Apple, ExecuTorch and ONNX for mobile graphs, WebGPU for zero-install previews.