TPEE Toptronic® Prompt Engineering Editor

97. Quantization 3 - Advanced Methods and Calibration

Compare GPTQ, AWQ, GGUF k-quants and quantization-aware training, and know when calibration data decides the result.

By Jacques Botte, founder of Toptronic®. Last updated 17 October 2026.

The lesson

GPTQ quantizes layer by layer and uses a small calibration set to choose each weight's rounded value so that the layer's output changes as little as possible. AWQ starts from a different observation: activation magnitudes, not weight magnitudes, identify which channels matter, so it protects those channels and quantizes the rest harder. Both produce 4-bit GPU weights; both depend on the calibration data resembling what you will actually run.

GGUF is a file format, not a single method. Its k-quant recipes mix precision inside one model: important tensors stay at higher bits, less important ones drop lower, and the suffix encodes the mixture. Q4_K_M and Q5_K_M are the common middle ground; Q2 and Q3 exist and are visibly degraded. The practical advantage of GGUF is that one file carries everything a local runner needs, including the tokenizer and metadata.

Calibration is where these methods quietly differ. The calibration set is typically a few hundred short passages from a general corpus. If your workload is code and the calibration set is prose, the quantized model can lose more code accuracy than the headline number suggests. The honest procedure is to quantize, then evaluate on your own task, not to trust a table measured on someone else's benchmark.

Activation quantization is a further step beyond weights: the intermediate values computed during a forward pass are also stored in low precision. It is harder because activations have wider dynamic range and are computed at run time, so the scale must be selected online. FP8 formats with per-tensor scaling have made this practical on recent hardware, and W8A8 configurations are common in serving stacks that want throughput without a big quality trade.

Quantization-aware training sits at the other end: the model is trained with the rounding simulated, so it learns weights that survive the rounding. It costs a training run and produces the best low-bit results, which is why the research frontier leans on it. For anyone not training, the rule is simpler: prefer a well-known recipe at 4 or 5 bits, evaluate on your own examples, and treat every quality claim as a hypothesis until measured on your data.

Check yourself

Question 1: What does AWQ use to decide which channels to protect?
  1. The size of the weights only
  2. Activation magnitudes, which identify the channels that matter — correct
  3. The tokenizer frequency table
  4. The order of the layers

Answer: Activation magnitudes, which identify the channels that matter

AWQ is activation-aware: it protects channels whose activations are large, rather than ranking by weight size.

Question 2: What is GGUF?
  1. A quantization algorithm owned by one vendor
  2. A single-file model format whose k-quant recipes mix precision inside one model — correct
  3. A GPU driver interface
  4. A method of training

Answer: A single-file model format whose k-quant recipes mix precision inside one model

GGUF is a container format. The Q4_K_M style suffixes describe which k-quant recipe was used inside it.

Question 3: Why can a 4-bit model calibrated on prose lose more accuracy on code than the published table suggests?
  1. Code tokenizes differently in a way quantization cannot see
  2. The calibration data did not resemble the workload, so the chosen rounding suits prose — correct
  3. Code models cannot be quantized
  4. The KV cache is larger for code

Answer: The calibration data did not resemble the workload, so the chosen rounding suits prose

Calibration sets the rounding decisions. A mismatched calibration set produces a model tuned for the wrong distribution.

← Previous lesson · All 106 lessons · Next lesson →

The full course — 106 lessons and 318 quiz questions — ships inside the app. Get TPEE to study it offline.