TPEE Toptronic® Prompt Engineering Editor

100. Latency 1 - Getting Started

Separate the three latencies users feel, know what drives each, and measure them instead of guessing.

By Jacques Botte, founder of Toptronic®. Last updated 17 October 2026.

The lesson

Latency is how long a person waits, and for a language model that wait has three parts. Time to first token is how long before anything appears, driven mostly by how much input the model must read. Then generation speed, usually quoted as tokens per second, decides how fast the answer streams. Then everything else: network hops, queueing on the provider's side, your own application's work before and after the call. A single 'latency' number hides all of this.

The input side is called prefill, and it processes the whole prompt at once. That makes it fast per token but it scales with prompt length, so a very long context costs real time before the first word arrives. The output side is called decode, and it produces one token at a time, each requiring a full pass through the model. Decode is why a long answer feels slow even when the model is quick to start.

The dominant driver of decode speed is memory bandwidth, not arithmetic. Every generated token requires reading the model's weights through the memory bus, so a smaller or more heavily quantized model generates faster on the same hardware, and a graphics card with more bandwidth beats one with more compute for this particular job. This is the same fact that makes quantization worthwhile, seen from the other end.

For agents the numbers compound. One user turn can become ten or twenty model requests, each with its own time to first token and its own decode, plus the time the tools take. A model that feels instant in a chat window can make an agent feel broken, because the user is waiting for twenty of those instants in a row. When an agent is slow, measure the number of round trips before blaming the model.

Measurement is the only way to know. Time the first token separately from the rest, run each case more than once because the first call is always slower, and record the distribution rather than the average. A single fast run and a single slow run tell you nothing about what a user will experience on a normal afternoon. TPEE's own windows are local and instant by design; every lesson you read here comes from a file on this disk.

Check yourself

Question 1: Which two numbers describe the wait a user feels from a language model?
  1. Download size and install time
  2. Time to first token and tokens per second — correct
  3. Parameter count and bit width
  4. Temperature and top-p

Answer: Time to first token and tokens per second

Time to first token covers prefill; tokens per second covers decode. Together they describe the felt wait.

Question 2: What usually limits token generation speed?
  1. Disk read speed
  2. Memory bandwidth, because the weights are read for every token — correct
  3. The tokenizer's vocabulary size
  4. The screen refresh rate

Answer: Memory bandwidth, because the weights are read for every token

Decode is bandwidth-bound, which is why smaller or quantized weights generate faster on the same hardware.

Question 3: Why can an agent feel far slower than a chat window using the same model?
  1. Because agents use a larger tokenizer
  2. Because one user turn becomes many sequential model requests plus tool time — correct
  3. Because agents disable caching
  4. Because agents run offline

Answer: Because one user turn becomes many sequential model requests plus tool time

Latency compounds across round trips. Counting requests is the first diagnostic when an agent feels slow.

← Previous lesson · All 106 lessons · Next lesson →

The full course — 106 lessons and 318 quiz questions — ships inside the app. Get TPEE to study it offline.