Guide

Core optimizations

Activation outliers

A few channels with huge magnitudes set a single INT8 scale and squash everyone else. LLM.int8() and SmoothQuant exist to stop that.

61 of 138 · 2 min read

LLM activations have systematic outliers: a few channels with huge magnitudes. A single per-tensor INT8 scale dictated by those outliers squashes everyone else into a few bins. The result is noise, and the model looks mysteriously dumb.

This is why naive "just INT8 the activations" fails on LLMs, and why weight-only recipes were popular before SmoothQuant and FP8 paths matured.

Concept

An outlier channel is a hidden dimension whose activations are far larger than the rest. One scale that covers that channel wastes most of the integer range on empty space.

The big idea

Think of photographing a room with one open window. If the camera meters on the window, the room goes black. If it meters on the room, the window blows out. You need either two exposures or a way to dim the window.

The picture breaks if you only look at weights. Weight distributions are usually better behaved. The hard part is often the activations.

Walking through the two mitigations

LLM.int8() (bitsandbytes): keep outlier-associated parts in FP16, quantize the rest to INT8. The matmul is mixed precision. You pay a little extra compute for the outlier slice and keep quality.

SmoothQuant: migrate difficulty from activations to weights with per-channel smoothing scales, then quantize both more evenly. That enables practical W8A8.

smooth:  X' = X / s     W' = W × s
then:    quantize X' and W' onto INT8 grids

The product X W is unchanged in exact arithmetic. The scales make X' less peaky and W' a bit wider, which INT8 can stand.

If an INT8 deployment looks "mysteriously dumb," check whether you quantized activations without an outlier strategy.

Common mistakes

  • One per-tensor activation scale. The outliers own the range. Everyone else gets two or three bins.
  • Blaming GPTQ when the failure is W8A8 without SmoothQuant. Weight-only INT4 and activation INT8 fail for different reasons.
  • Skipping a task eval. Perplexity can look fine while tool-calling JSON breaks. Logit noise hits structured output first.

Check yourself

  1. Why does one huge channel ruin per-tensor INT8? Answer: the scale must cover that channel, so typical values collapse into a few integer bins.
  2. What does SmoothQuant move, and why does the matmul stay the same? Answer: it divides activations by s and multiplies weights by s. The product X W is unchanged.

Key takeaways

  • Activation outliers break naive INT8 on LLMs.
  • LLM.int8() keeps the outlier slice in FP16.
  • SmoothQuant moves difficulty onto weights so both sides can be INT8.
  • If INT8 looks dumb, check the outlier strategy before the calibration set.

Further learning

Next, the weight-only workhorses: GPTQ and AWQ.

Quantization

INT8, FP8, INT4 and FP4. Fewer bits shrink weights, activations and KV. Match the format to memory-bound decode or compute-bound prefill, then eval quality.

Weight-only vs activation quantization

W4A16 stores 4-bit weights and keeps 16-bit activations. W8A8 and FP8 quantize both sides so Tensor Cores run native low-precision matmuls.

GPTQ and AWQ

Two post-training methods that compress a finished checkpoint to INT4 with a small calibration set. GPTQ uses second-order updates. AWQ protects salient channels.