Edge runtimes and GGUF

The bandwidth napkin tells you if a model can feel fast. The runtime tells you whether it will actually load, quantize and run on the devices you ship. Match the stack to the OS matrix. Do not pick a format because a demo looked good on one laptop.

The big idea

A shipping container that only fits one harbor is a bad container. llama.cpp and GGUF are the cross-platform container for local LLMs. Apple-first products should still prefer MLX or Core ML, because those stacks are built around unified memory and the Neural Engine.

Walking through the stacks

llama.cpp popularized local LLMs: C/C++ runtime, CPU/Metal/CUDA/Vulkan backends, and GGUF, which packs weights, tokenizer and metadata for mmap-friendly loading.

Quant types are a dial, not a single "4-bit":

Pick the largest quant that fits comfortably in RAM with headroom for KV and the OS. Measure task evals. Do not assume all "4-bit" labels are equal.

MLX is Apple's array framework with first-class unified-memory LLM examples. On M-series machines it is often the fastest path for supported models. Prefer it for Mac-first products. Keep llama.cpp when you need one binary across Linux, Windows, Mac and discrete GPUs.

• Core ML — ship on iOS/macOS with Neural Engine offload. Convert carefully. Watch op coverage.
• ExecuTorch — PyTorch-native edge runtime for mobile and embedded. Growing LLM support via delegates.
• ONNX Runtime — strong for encoders, classifiers, speech frontends and classic nets. The LLM story varies by Execution Provider.

WebLLM / WebGPU downloads a quantized model into the browser cache and runs on WebGPU. UX win: no install, clear privacy story. Cost: large downloads, GPU variance, tab-lifetime memory limits. Good for demos, extensions and "try locally" modes — not for 70B.

NPUs (Apple Neural Engine, Qualcomm Hexagon) shine on fixed graphs and quantized matmuls. Vendor stacks matter as much as raw TOPS marketing numbers.

Common mistakes

• One "4-bit" label. Q4_K_M and a desperate Q2_K are not the same quality.
• MLX for a Windows/Linux fleet. Use llama.cpp when you need one binary.
• 70B in the browser. Downloads, memory limits and GPU variance will sink it.

Check yourself

1. You need one desktop binary for Linux, Windows and Mac. Which runtime? Answer: llama.cpp / GGUF.
2. Why is Q4_K_M a common laptop default? Answer: about 4.5 bits, a quality/size sweet spot that still leaves RAM for KV and the OS.

Key takeaways

• llama.cpp/GGUF for cross-platform desktop. MLX or Core ML for Apple-first.
• ExecuTorch and ONNX for mobile encoders and constrained graphs. WebGPU for previews.
• Lock the quant type in CI with a tiny eval set.

Further learning

Next, keep quality usable with hybrid on-device routing.