Guide

Core optimizations

Weight-only vs activation quantization

W4A16 stores 4-bit weights and keeps 16-bit activations. W8A8 and FP8 quantize both sides so Tensor Cores run native low-precision matmuls.

60 of 138 · 2 min read

Two serving recipes get confused because both say "quantized." Weight-only stores weights in 4- or 8-bit and keeps activations in 16-bit. Weight-plus-activation puts both sides in low precision so Tensor Cores run native low-precision matmuls.

The first helps memory-bound decode. The second helps compute-bound prefill and large batches. Pick from the roofline, not from the checkpoint file name.

Concept

W4A16 means 4-bit weights, 16-bit activations. W8A8 means 8-bit weights and 8-bit activations. FP8×FP8 is the floating-point cousin of W8A8.

The big idea

Weight-only is like keeping a compressed dictionary on the shelf and expanding each word as you read it. You moved fewer bytes off the shelf. The reading itself is still in the original language.

Weight-plus-activation is like doing the arithmetic in the compressed language. That needs hardware that speaks it, and it needs activations that survive the coarse grid.

The picture breaks if you dequantize to FP16 before every matmul. Then you paid for a small file and threw the speed win away.

Walking through the two recipes

Weight-only (W4A16, W8A16): kernels dequantize weights in registers or SRAM while multiplying. Great for decode, which is limited by HBM traffic. Prefill gains are smaller if compute was already the limiter and dequant overhead eats the win.

Weight-plus-activation (W8A8, FP8×FP8): both sides low precision. Helps prefill and large batches. Needs activation quantization that survives outliers.

Trade-off

Prefer W4A16 (or W4A8 hybrids) when you are GPU-memory poor or decode-latency bound on older kernels. Prefer FP8 (or W8A8) on Hopper and Blackwell when you want a balanced serving default with strong Tensor Core use and a simpler quality profile than aggressive INT4.

Example

Compress a 70B BF16 checkpoint (~140 GB weights) with AWQ W4A16 group 128 to about 35 GB. On an 80 GB GPU you now fit weights plus much more KV. Decode tokens/s climb because each step reads about 4× fewer weight bytes, kernel-dependent.

Common mistakes

  • Shipping a 4-bit file that the engine unpacks to FP16. That is a storage trick, not an inference win.
  • Using W4A16 to speed a compute-bound prefill. The dequant can eat the gain. Try FP8.
  • Using FP8 on hardware without a native path. You are back to unpacking.

Check yourself

  1. Decode is memory-bound and HBM is tight. Which recipe do you try first? Answer: weight-only INT4 or FP8 weights, so each step moves fewer bytes.
  2. Prefill is compute-bound on Hopper. Why might FP8 beat W4A16? Answer: FP8 Tensor Cores do the matmul in low precision. W4A16 may dequant to 16-bit and stay on the slower path.

Key takeaways

  • W4A16 helps bandwidth-bound decode. W8A8 and FP8 help Tensor Cores.
  • A quantized file is not a win unless the runtime GEMM uses that format.
  • Default to FP8 on modern NVIDIA servers. Use INT4 when memory or edge demands it.
  • Activation quantization needs an outlier strategy.

Further learning

Next, why naive INT8 fails: activation outliers.

Quantization

INT8, FP8, INT4 and FP4. Fewer bits shrink weights, activations and KV. Match the format to memory-bound decode or compute-bound prefill, then eval quality.

Quantization formats and scales

How many bits a weight uses, and how scales stretch a coarse grid over a tensor. Per-tensor, per-channel and per-group scales trade overhead for quality.

The roofline model and arithmetic intensity

Predicting performance from FLOPs and bytes: compute-bound vs memory-bound. Start here for the map, then work the examples.