Guide

Core optimizations

Quantization formats and scales

How many bits a weight uses, and how scales stretch a coarse grid over a tensor. Per-tensor, per-channel and per-group scales trade overhead for quality.

58 of 138 · 3 min read

Full-precision weights are essentially continuous 16-bit floats. Quantization snaps each weight, or activation, onto a coarse grid. Scales stretch that grid to cover the tensor's range. Group-wise scales let different regions pick different stretches. Error is the distance from the original value to the nearest grid point, then propagated through the network.

From numbers and text you already know floats trade exponent for mantissa. FP8 E4M3 vs E5M2 is the same trade in eight bits. INT4 has no exponent at all, only a scaled integer, which is why group scales matter so much.

Concept

A bit is a binary digit. Cutting from 16-bit to 4-bit weights is a 4× reduction in weight bytes. Dynamic range and precision both shrink. Scales and smarter formats recover usable accuracy.

The big idea

Think of a ruler. A 16-bit ruler has fine marks. A 4-bit ruler has 16 marks. A scale stretches those 16 marks over the range you care about. One scale for a whole matrix is a blunt stretch. A scale every 128 weights is a set of short rulers.

The picture breaks when one huge value sets the stretch for everyone else. That is the outlier problem.

Walking through the formats

FormatBitsNotes
FP3232Training default historically; rare for LLM inference
FP16 / BF1616BF16 preferred for training stability; common inference baseline
FP8 E4M38More fraction bits, better for weights and activations on Hopper and later
FP8 E5M28More exponent range, often gradients or some activation paths
INT88Uniform integers plus a scale, and an optional zero-point
INT44Weight-only staple (GPTQ/AWQ); needs good grouping
FP4 / MXFP4 / NVFP44Emerging low-precision floats on newer NVIDIA stacks

Integer quantization maps real values with an affine transform, for example real ≈ scale × (q − zero_point). Symmetric schemes often fix zero-point at 0 and only store scale.

Granularity controls how many scales you store:

  • Per-tensor: one scale for a whole weight matrix. Cheap, blunt.
  • Per-channel / per-token: scales along an axis. Much better accuracy.
  • Per-group (for example group size 128): a scale every 128 weights. The practical INT4 default.

Finer granularity means more scale overhead and better quality. Group 128 is the usual compromise.

70B in BF16:  70e9 × 2 = 140 GB
70B in FP8:   70e9 × 1 =  70 GB
70B in INT4:  70e9 × 0.5 = 35 GB

Common mistakes

  • Reading "4-bit on the hub" as a serving win. If the runtime unpacks to FP16 before the matmul, you only saved disk.
  • Using one per-tensor scale for INT4. Quality usually collapses. Use group 128 or finer.
  • Mixing E4M3 and E5M2 without a reason. They trade fraction bits for range. Match the tensor.

Check yourself

  1. How many bytes do 70 billion parameters take in INT4? Answer: 70e9 × 0.5 = 35 GB.
  2. Why does group size 128 usually beat one scale per matrix? Answer: each group picks its own stretch, so a few large weights do not squash the rest.

Key takeaways

  • Quantization snaps values onto a grid. Scales stretch the grid to the tensor.
  • Know BF16, FP8 (E4M3/E5M2), INT8, INT4, and tensor vs channel vs group scales.
  • Group 128 is the usual INT4 compromise.
  • The format only speeds serving if the kernel uses it.

Further learning

Next, weight-only vs activation quantization. For a visual tour, see A Visual Guide to Quantization.

Quantization

INT8, FP8, INT4 and FP4. Fewer bits shrink weights, activations and KV. Match the format to memory-bound decode or compute-bound prefill, then eval quality.

Numbers, text and encodings

How integers, floats, text, timestamps and Base64 are really stored as bytes, and the production bugs each one causes. Start here for the data-representation map.

Inference arithmetic

Napkin math for any model on any GPU: weight memory, decode and prefill time, batch limits and KV budget.