TPEE Toptronic® Prompt Engineering Editor

104. Model Weights 3 - Precision, Memory and Bandwidth

Reason about where weights live during inference, and why bandwidth rather than compute decides generation speed.

By Jacques Botte, founder of Toptronic®. Last updated 17 October 2026.

The lesson

Precision is a per-tensor decision, not a per-model one. A model may keep its attention projections at BF16, its feed-forward weights at FP8 and its KV cache at FP8 as well. Each choice trades bytes against error in a place where error costs differently. Understanding that layers tolerate precision unevenly is what makes a mixed configuration reasonable rather than arbitrary.

Where the weights live matters as much as how big they are. GPU memory is fast but small; system memory is large but roughly an order of magnitude slower; NVMe is slower again. Offloading some layers to system memory lets a model that does not fit still run, at a heavy speed cost, because every token's forward pass now crosses a slow bus. The usual symptom is a model that works and is unusably slow, with the graphics card idle.

During generation the model reads essentially all of its weights for every token. That makes decode bandwidth-bound: the time per token is roughly the bytes that must move divided by the achievable bandwidth. Doubling batch size amortises the same reads over more tokens, which is why throughput rises and why a single-user latency figure and a server throughput figure are not comparable.

Prefill behaves oppositely. Reading a long prompt is compute-bound because the same weights are reused across many positions, so arithmetic units, not the bus, set the time. This is why a long prompt hurts time to first token far more than it hurts tokens per second, and why batching helps prefill disproportionately.

The practical conclusions are short. Choose the smallest precision that passes your own evaluation, because bytes moved is time and money. Keep as much of the model resident as fits, because offload costs bandwidth. Measure generation and prefill separately, because they are limited by different resources. And never compare two models on parameter count alone: shapes, precision and memory layout decide what actually happens on your hardware.

Check yourself

Question 1: What limits token generation speed for a single request?
  1. Arithmetic throughput
  2. Memory bandwidth, because the weights are read per token — correct
  3. Disk latency
  4. The tokenizer

Answer: Memory bandwidth, because the weights are read per token

Decode is bandwidth-bound; prefill is the compute-bound phase.

Question 2: Why does offloading layers to system memory hurt so much?
  1. Because system memory is smaller
  2. Because every token's forward pass then crosses a much slower bus — correct
  3. Because the weights are re-quantized
  4. Because the KV cache is deleted

Answer: Because every token's forward pass then crosses a much slower bus

The cost is bytes over a slower interface, paid on every generated token.

Question 3: Why does a long prompt hurt time to first token more than generation speed?
  1. Because prefill is compute-bound and scales with prompt length — correct
  2. Because the model must be reloaded
  3. Because the vocabulary grows
  4. Because caching is disabled

Answer: Because prefill is compute-bound and scales with prompt length

Prefill processes all input positions; its cost rises with prompt length, while decode cost per token stays roughly flat.

← Previous lesson · All 106 lessons · Next lesson →

The full course — 106 lessons and 318 quiz questions — ships inside the app. Get TPEE to study it offline.