95. Quantization 1 - Getting Started

The lesson

Quantization is storing a model's numbers in fewer bits. A model is a very large collection of weights, and by default those weights are usually 16-bit floating-point numbers, two bytes each. Quantization rewrites them as 8-bit or 4-bit values, so the same model occupies a quarter or an eighth of the memory and moves through the memory bus in a quarter or an eighth of the time. Nothing about the model's structure changes: it still has the same layers, the same attention, the same behaviour. Only the precision of each stored number changes.

The reason anyone cares is arithmetic. A 7-billion-parameter model at 16 bits needs about 14 GB just to hold its weights, before you add the space for the conversation itself. At 4 bits it needs about 3.5 to 4 GB. That is the difference between a model that will not fit on a consumer graphics card and one that will. It is also the difference between a cloud bill you can afford and one you cannot, because providers charge for the memory and compute they reserve.

The cost is accuracy, but not evenly. Most weights tolerate being rounded coarsely; a small number of them carry outsized influence, and rounding those badly damages output. Every real quantization method spends its effort on that problem: it keeps a scale factor per group of weights so the rounding error stays proportional, and it protects the outliers that matter. A well-made 4-bit model is usually close to its 16-bit original on everyday tasks and measurably worse on the hardest reasoning. A careless one is worse everywhere.

In practice you will meet three families. GGUF files with names like Q4_K_M are for local runners and mix precision across the model. GPTQ and AWQ are weight-quantization methods used for GPU serving. Bitsandbytes does 8-bit and 4-bit at load time with almost no preparation. The naming tells you the bit width and the recipe; Q4_K_M means roughly four bits per weight with a k-quant scheme and medium quality settings.

Where TPEE fits: nowhere at runtime, deliberately. This app never downloads, quantizes or runs a model. It teaches you the vocabulary so that when you choose a local model in a separate tool, you can read the file name and know what you are trading. The lesson exists because the most common beginner mistake is downloading a 16-bit model onto a machine that can only run the 4-bit one, concluding that local AI is slow, and giving up.

Check yourself