98. Quantization 4 - Professional Serving Decisions

The lesson

In production the question is never 'which quantization is best' but 'which quantization for this task at this budget'. Start by fixing the constraint: is it memory (must fit one card), latency (must answer in two seconds), throughput (must serve a thousand requests an hour), or cost per request? Each points at a different setting, and optimizing the wrong one is how teams ship a model that is cheap and unusable.

Then build a small evaluation set from your own traffic: fifty to two hundred real prompts with known good answers, scored by a rule where possible and by a human where not. Run the 16-bit model through it to establish the ceiling, then each candidate quantization. Report the delta, not the absolute score. A 4-bit model that loses one point on a translation task is a bargain; the same loss on a strict JSON-extraction task may be unacceptable because failures are silent.

Watch the failure shape, not only the rate. Quantization damage rarely looks like a random wrong answer; it looks like longer rambling, weaker instruction-following at the edges, and more frequent format violations. For structured output that means more parse errors downstream, which cost more than the memory you saved. For free text it may be invisible. Measure the thing you actually consume.

Serving hardware decides more than the bit width. A quantized model whose kernels are not optimized for your GPU can be slower than a 16-bit model with good kernels, because dequantization work and memory access patterns dominate. Memory bandwidth, not arithmetic, is the usual limit during token generation, and that is precisely why smaller weights help: fewer bytes move per token. If your stack spends its time dequantizing badly, a supported 8-bit format beats an exotic 4-bit one.

Finally, record the decision. Write down the model, the exact file or quant recipe, the hardware, the evaluation set, the measured delta and the date. Models and stacks change monthly, and a configuration chosen six months ago on a now-unsupported kernel is a silent regression waiting to be rediscovered. This is the same discipline the rest of this course applies to prompts: a number you cannot reproduce is not evidence.

Check yourself