TPEE Toptronic® Prompt Engineering Editor

101. Latency 2 - Expert

Reason about percentiles, batching, prefill versus decode limits and caching, and design a latency budget you can defend.

By Jacques Botte, founder of Toptronic®. Last updated 17 October 2026.

The lesson

Averages are useless for latency. Users remember the slow request, so the figures that matter are p50, p95 and p99, and the gap between them is the whole story. A service with a 400 ms p50 and a 6 s p99 is not a fast service with an occasional hiccup; it is a service whose worst decile is broken, and in an agent that runs twenty calls per turn the p99 is what the user meets far more often than 1 in 100.

Batching is the central trade. Serving stacks group concurrent requests so the weights are read once for many tokens, which raises throughput enormously and raises the latency of each individual request. If you are tuning a system, know which side you are on: a batch of one is bandwidth-bound and latency-optimal, a large batch is compute-bound and throughput-optimal. Claims about tokens per second mean nothing without the batch size and the concurrency they were measured at.

Prefill and decode fail differently. Prefill is compute-heavy and scales with prompt length, so a huge retrieved context or a long system prompt delays the first token. Decode is bandwidth-heavy and scales with output length. Newer stacks split prefill into chunks and interleave it with ongoing decodes so one long prompt cannot freeze everyone else; that improves p99 at the cost of a slightly later first token for the long request itself.

Caching changes the arithmetic more than any knob. A prefix cache means the repeated part of a prompt, usually the system prompt and tools, is not reprocessed, so the first-token time drops sharply and cost falls with it. Break the shared prefix, for example by injecting a timestamp at the top, and every request pays full prefill again. If latency regressed without a code change on your side, look for something that made the prefix unique.

Design the budget explicitly. Decide the acceptable p95 for first token and for a full answer, then allocate it: network, queue, prefill, decode, tools, your own post-processing. Write down what happens when the budget is exceeded, which is usually a shorter answer, a smaller model, or an honest progress indicator. Little's law is the sanity check that ties it together: concurrency equals arrival rate times time in system, so a system at its limit does not degrade gracefully, it queues.

Check yourself

Question 1: Which latency figure best predicts what users complain about?
  1. The mean
  2. p50
  3. The p95 and p99 tail — correct
  4. The fastest observed run

Answer: The p95 and p99 tail

Tail latency is what a user meets repeatedly across the many calls of an agent turn.

Question 2: What is the throughput-versus-latency trade in batching?
  1. Batching raises both throughput and per-request latency — correct
  2. Batching lowers both
  3. Batching only affects cost
  4. Batching has no effect on latency

Answer: Batching raises both throughput and per-request latency

A larger batch reads the weights once for many tokens, raising throughput while each request waits longer.

Question 3: What happens when a shared prefix cache is broken by, for example, a timestamp in the system prompt?
  1. Nothing measurable
  2. Every request re-pays full prefill, so first-token latency and cost rise — correct
  3. The model becomes more accurate
  4. The KV cache is deleted

Answer: Every request re-pays full prefill, so first-token latency and cost rise

Prefix caching depends on an exact shared prefix; a unique leading token invalidates it for every request.

← Previous lesson · All 106 lessons · Next lesson →

The full course — 106 lessons and 318 quiz questions — ships inside the app. Get TPEE to study it offline.