TPEE Toptronic® Prompt Engineering Editor

99. Quantization 5 - Expert

Understand where the research frontier is going, and audit a quantization claim the way you would audit a benchmark.

By Jacques Botte, founder of Toptronic®. Last updated 17 October 2026.

The lesson

The frontier has moved from 4 bits toward formats that were considered unusable a few years ago. FP8 with per-tensor or per-block scaling is now standard on recent accelerators for both weights and activations. Microscaling formats push to FP4 and FP6 with a shared scale per small block, and hardware support is arriving with them. Ternary and 1.58-bit research networks show that very low precision can work when the model is trained for it rather than converted after the fact.

That last distinction is the important one. Post-training quantization converts a finished model and hopes the damage is small. Quantization-aware training, and training natively in low precision, lets the model adapt. Results that look impossible for post-training conversion (2-bit weights that still answer questions) usually come from training-time methods, and quoting them as evidence that '2-bit is fine now' is a category error.

KV-cache precision is its own research area because the cache, not the weights, dominates memory at long context. FP8 KV caches are widely deployed; 4-bit caches with per-channel scales are an active area. The trade is subtle: the cache is read for every generated token, so a lower-precision cache both saves memory and reduces bandwidth, but errors accumulate across the whole context rather than being applied once.

There is also an interaction with the rest of the inference stack. Speculative decoding uses a small draft model to propose tokens and a large model to verify them; quantizing the draft and the verifier differently changes the acceptance rate, so a configuration that was fast at 16 bits can be slower after quantization. Batching changes the balance too, because a large batch makes the run compute-bound while a batch of one is bandwidth-bound. Any serious throughput claim names its batch size.

The expert habit is audit. When someone claims a quantized model matches the original, ask: which evaluation, how many items, what scoring rule, what calibration data, what hardware, what batch size, and was the 16-bit baseline run on the same stack? Then run your own fifty examples. A quantization decision that survives that scrutiny is engineering; one that does not is a number on a slide, and this course has already taught you what to do with those.

Check yourself

Question 1: Why is it a category error to cite 2-bit results from natively-trained low-precision models as proof that post-training 2-bit conversion is safe?
  1. Because 2-bit models are always worse
  2. Because those models were trained to survive the rounding, not converted after training — correct
  3. Because post-training conversion does not exist
  4. Because 2-bit weights cannot be stored

Answer: Because those models were trained to survive the rounding, not converted after training

Training-time low precision lets the model adapt to the constraint; post-training conversion does not.

Question 2: Why does KV-cache quantization matter more than weight quantization at long context?
  1. Because the cache is written to disk
  2. Because the cache can exceed the weights in memory and is read on every generated token — correct
  3. Because the cache holds the tokenizer
  4. Because the cache is trained

Answer: Because the cache can exceed the weights in memory and is read on every generated token

At long context the KV cache dominates memory and is re-read per token, so its precision affects both memory and bandwidth.

Question 3: Which question belongs in an audit of a quantization claim?
  1. What is the model's marketing name?
  2. Which evaluation, how many items, what calibration data, which hardware and batch size? — correct
  3. How large is the download?
  4. How many stars does the repository have?

Answer: Which evaluation, how many items, what calibration data, which hardware and batch size?

A quality or throughput claim is only meaningful with the evaluation, the data, the hardware and the batch size attached.

← Previous lesson · All 106 lessons · Next lesson →

The full course — 106 lessons and 318 quiz questions — ships inside the app. Get TPEE to study it offline.