Core optimizations
Quantization
INT8, FP8, INT4 and FP4. Fewer bits shrink weights, activations and KV. Match the format to memory-bound decode or compute-bound prefill, then eval quality.
A 70B model in BF16 is roughly 140 GB of weights alone (70e9 × 2 bytes). That is multiple high-end GPUs before you store a single KV entry. Cut weights to 4 bits and the same parameters fit in about 35 GB, often one GPU, with room for KV. Decode, being memory-bound, also gets faster: fewer bytes to stream from HBM per token.
The cost is numerical error. Done carelessly, you tank reasoning and long-context quality. Done carefully, loss is small enough that product prefers the cheaper token.
Why quantize
From inference arithmetic:
- Memory capacity: smaller weights mean fewer GPUs, a larger KV budget, a higher batch.
- Memory bandwidth (decode): weight-only INT4 or FP8 means less HBM traffic per token.
- Compute (prefill / large batch): INT8 and FP8 Tensor Core paths can raise throughput, but only if kernels use those units.
Concepts in this section
| Step | Concept | What it answers |
|---|---|---|
| 1 | Formats and scales | Which grids exist, and how do scales work? |
| 2 | Weight-only vs activations | W4A16 for decode, or FP8 for Tensor Cores? |
| 3 | Activation outliers | Why does naive INT8 fail on LLMs? |
| 4 | GPTQ and AWQ | How do you build an INT4 checkpoint? |
| 5 | KV cache quantization | How do you shrink the live cache? |
Practical defaults
- Hopper / Blackwell serving: FP8 weights, and often activations and KV, when the engine and hardware support it cleanly.
- Memory-constrained or edge: INT4 weight-only (AWQ/GPTQ) or GGUF-style quants for on-device.
- Always validate on your traffic. Prefer per-channel or group scales over per-tensor for INT schemes.
- Treat "4-bit on Hugging Face hub" as untrusted until you know the method, group size and evals.
Checklist when you compress a model:
- Pick a format family from hardware.
- Compress with a known recipe. Record calibration set and seed.
- Confirm the runtime kernel matches the file (Marlin, Cutlass FP8, and similar).
- Recompute memory budgets. Freed HBM should become KV, not idle bytes.
- Eval: perplexity smoke test, then task suite, then long-context and reasoning. Keep a BF16 teacher.
- Load test TTFT, TPOT and quality under concurrency. Numerics can change early stopping.
Trade-off
One big FP8 model vs two INT4 models on one GPU. FP8 usually preserves quality better and simplifies ops. Two INT4 replicas can raise QPS if quality holds. Prefer FP8 until memory forces packing.
Common mistakes
- Shipping on WikiText perplexity while production is tool-calling JSON.
- Quantizing then unpacking to FP16 in the runtime. Disk shrank. The server did not get faster.
- Mixing dtypes across tensor-parallel ranks. Quantize, then shard on the engine's supported path.
Key takeaways
- Quantization cuts bytes to fit models and to speed memory-bound decode.
- Compute-bound wins need real low-precision kernels.
- W4A16 helps bandwidth. W8A8 and FP8 help Tensor Cores.
- GPTQ and AWQ are the INT4 workhorses. Eval long-context and reasoning, not only perplexity.
- Default to FP8 on modern NVIDIA servers. INT4 when memory or edge demands it.
Further learning
A Visual Guide to Quantization (Maarten Grootendorst)Exploring memory-efficient techniques for LLMsnewsletter.maartengrootendorst.com
Related
Inference arithmetic
Napkin math for any model on any GPU: weight memory, decode and prefill time, batch limits and KV budget.
Numbers, text and encodings
How integers, floats, text, timestamps and Base64 are really stored as bytes, and the production bugs each one causes. Start here for the data-representation map.
On-device and edge inference
Run models on laptops, phones and browsers when latency, privacy or offline matter — with napkin math for bandwidth-bound decode and the right runtime.