TPEE Toptronic® Prompt Engineering Editor

96. Quantization 2 - Number Formats and Memory Maths

Read a quantization label, do the memory arithmetic yourself, and understand what a scale factor is doing.

By Jacques Botte, founder of Toptronic®. Last updated 17 October 2026.

The lesson

Four number formats explain almost every label you will see. FP32 uses four bytes per weight and is the training default. FP16 and BF16 use two bytes; BF16 keeps the same exponent range as FP32 with fewer mantissa bits, which is why models prefer it, because range matters more than fine detail during training. INT8 uses one byte and INT4 half a byte. A quantization label is simply which of these the weights were stored as, plus the recipe used to get there.

The memory arithmetic is worth doing by hand once. Weight memory in bytes equals parameter count times bytes per weight. A 70-billion-parameter model is 140 GB at 16 bits, 70 GB at 8 bits and 35 GB at 4 bits. Add the KV cache, which grows with context length and batch size and is not quantized by default, plus the runtime's own overhead. When someone says a model 'needs 48 GB', they usually mean weights plus a realistic KV cache plus a margin.

A quantized weight is not simply a rounded number; it is an integer plus a scale. A block of, say, 32 or 128 weights shares one floating-point scale, and the stored integers are multiplied by it at load or at compute time. This is why group size appears in method names: a smaller group means a scale that tracks local variation more closely and better accuracy, at the cost of a few extra bytes per group and slightly more work in the kernel.

Outliers are the enemy. A few weights in each layer are much larger than the rest, and a single shared scale forced to cover them crushes all the small values toward zero. Good methods isolate those channels, keep them in higher precision, or rotate the data so no channel is extreme. That is the real difference between a method that claims 4 bits and a method that survives 4 bits.

The KV cache deserves its own sentence because it is the part beginners forget. It stores the keys and values for every token in the context so the model does not recompute them, and at long context it can exceed the weights. It can also be quantized, usually to 8 bits, with modest quality cost and a large memory saving. If your local model runs out of memory only on long conversations, the KV cache is the likely cause, not the weights.

Check yourself

Question 1: Weight memory in bytes is calculated as:
  1. parameter count divided by context length
  2. parameter count times bytes per weight — correct
  3. vocabulary size times layer count
  4. batch size times bytes per token

Answer: parameter count times bytes per weight

Bytes of weight memory = parameters x bytes per weight, before the KV cache and runtime overhead are added.

Question 2: Why is BF16 usually preferred over FP16 for model weights?
  1. It uses half the memory of FP16
  2. It keeps FP32's exponent range, so large and small values do not overflow or underflow — correct
  3. It is the only format GPUs support
  4. It compresses the tokenizer

Answer: It keeps FP32's exponent range, so large and small values do not overflow or underflow

BF16 trades mantissa precision for exponent range, which is the safer trade for deep networks.

Question 3: A local model only runs out of memory on very long conversations. What is the most likely cause?
  1. The weights changed size
  2. The KV cache grew with the context length — correct
  3. The tokenizer reloaded
  4. The graphics driver downgraded the model

Answer: The KV cache grew with the context length

Weights are fixed at load. The KV cache grows with every token in context, and it is the usual reason long chats fail where short ones worked.

← Previous lesson · All 106 lessons · Next lesson →

The full course — 106 lessons and 318 quiz questions — ships inside the app. Get TPEE to study it offline.