Core optimizations
GPTQ and AWQ
Two post-training methods that compress a finished checkpoint to INT4 with a small calibration set. GPTQ uses second-order updates. AWQ protects salient channels.
Post-training quantization (PTQ) takes a finished BF16 or FP16 checkpoint and a small calibration set, hundreds to thousands of examples. There is no full retraining. GPTQ and AWQ are the two methods you will actually meet for INT4 serving.
They spend their cleverness on where to put the grid and which weights deserve protection, not on changing the Transformer.
Concept
Calibration is a short run over real-looking text that tells the compressor the activation ranges and which weights matter. It is not training. The base weights stay frozen except for the rounding GPTQ applies.
The big idea
GPTQ is a careful accountant. It quantizes one column, then adjusts the remaining weights so the layer's output error stays small, using second-order (Hessian) information.
AWQ is a bodyguard. It looks at activation magnitudes, finds the weight channels that would hurt most if rounded, and scales those channels up before quantization so they keep more resolution.
The picture breaks when the calibration set does not look like production. A compressor tuned on Wikipedia will not protect the weights that matter for tool-calling JSON.
Walking through the choice
- GPTQ: quantize weights column by column. Adjust remaining weights to compensate error. Strong INT4. Slower to run than AWQ. Can push to very low bits.
- AWQ (Activation-aware Weight Quantization): identify salient channels from activation magnitudes, protect them by scaling before quantization. Typically fast calibration. Excellent INT4 quality for chat and code workloads.
Quantization-aware training (QAT) inserts fake-quant ops during training or finetune so the model learns to tolerate the grid. Higher effort. Used when PTQ quality is not enough: tight bit-width or sensitive domains.
As of when this was written, tooling has consolidated toward engine-friendly compressors, for example vLLM's llm-compressor ecosystem and GPTQModel. The algorithms to understand are still GPTQ, AWQ, and SmoothQuant.
Example
AWQ W4A16, group size 128, on a 70B BF16 checkpoint: weights drop from about 140 GB to about 35 GB. If coding pass@1 drops 2 points but cost per token drops 3×, product may accept it. If customer-facing legal summaries degrade, step up to FP8 or mixed precision.
A sane INT4 starting point is AWQ or GPTQ with group size 128. Record the calibration set and seed. Confirm the runtime kernel (Marlin, and similar) actually runs in 4-bit.
Common mistakes
- Shipping a hub "GPTQ" without group size, zero-points, or pack format. Load failures or silent quality cliffs.
- Calibrating on WikiText and shipping a tool-calling model. Structured outputs are brittle to logit noise.
- Treating QAT as the default. Try PTQ first. QAT is the escalation.
Check yourself
- You need INT4 tomorrow for a chat model. GPTQ or AWQ, and why? Answer: AWQ is the usual first try. Calibration is faster and quality is strong on chat and code.
- What does GPTQ do after it quantizes a column? Answer: it updates the remaining weights so layer output error stays small, using Hessian information.
Key takeaways
- PTQ compresses a finished checkpoint with a small calibration set.
- GPTQ compensates error column by column. AWQ protects salient channels.
- Group size 128 is a sane INT4 default. Record how you built the file.
- QAT is for when PTQ is not enough.
Further learning
AWQ and GPTQ. Next, KV cache quantization.
Related
Quantization
INT8, FP8, INT4 and FP4. Fewer bits shrink weights, activations and KV. Match the format to memory-bound decode or compute-bound prefill, then eval quality.
Quantization formats and scales
How many bits a weight uses, and how scales stretch a coarse grid over a tensor. Per-tensor, per-channel and per-group scales trade overhead for quality.
Activation outliers
A few channels with huge magnitudes set a single INT8 scale and squash everyone else. LLM.int8() and SmoothQuant exist to stop that.