118. Revisiting Context 5 - Measuring Whether Context Helped
Prove that the context you added earned its tokens: ablations, held-out questions and the arithmetic of a context change.
By Jacques Botte, founder of Toptronic®. Last updated 17 October 2026.
The lesson
Every context change is a hypothesis, and most of them are never tested. The document was added because it seemed relevant; the summary replaced the source because it was shorter; the examples were doubled because the output felt flat. Each of those is a claim about quality, cost or both, and each can be measured for less effort than people assume.
The instrument is an ablation. Take a set of real tasks with a known good answer, run them with the context and without it, and compare. Three outcomes matter: the change helped, it did nothing, or it hurt. The third is common and almost never noticed, because people add context and then only look at examples where it seemed to help. A context that helps on the cases you remember and hurts on the cases you do not is a net loss.
Measure cost on the same run. Tokens in, tokens out, latency, and money per task, before and after. A change that improves quality by two points and doubles the cost is a decision, not an improvement, and it should be made deliberately rather than discovered on an invoice. State the ratio you are willing to pay before you see the numbers, or the numbers will argue for themselves.
Watch for the two confounds that make context experiments lie. The FIRST is order: adding material at the top versus the middle changes the result for reasons that have nothing to do with the material. The SECOND is the judge: if the same model that produced the answer also scores it, a longer context will often score better because it looks more thorough. Use a different judge, or a deterministic check, or a person on a sample.
Finally, write the conclusion where the next person will find it: this context, for this task, measured this way, helped by this much at this cost, on this date. A note that says the manual was added to the prompt is a memory. A note that says the manual raised task success from 71 to 84 percent at 1.4 times the tokens, measured on thirty held-out questions, is engineering - and it is the only kind of note that survives the person who wrote it.
Check yourself
Question 1: What is the instrument for testing a context change?
- A longer prompt
- An ablation: the same real tasks with and without the context, compared — correct
- A second model
- A cost report
Answer: An ablation: the same real tasks with and without the context, compared
With, without, and compared - on tasks with a known good answer. Help, nothing, or harm are the three outcomes that matter.
Question 2: Why does adding material at the top versus the middle confound an experiment?
- Because position changes the result for reasons unrelated to the material — correct
- Because the top is smaller
- Because models ignore the top
- Because it changes the token count only
Answer: Because position changes the result for reasons unrelated to the material
Attention favours the beginning and the end. A position change is a separate variable and must be held constant or tested deliberately.
Question 3: A change improves quality two points and doubles the cost. What is it?
- An improvement
- A decision to be made deliberately, with the acceptable ratio stated in advance — correct
- A failure
- A defect
Answer: A decision to be made deliberately, with the acceptable ratio stated in advance
Quality per token is the real measure. State the ratio you will pay before you see the numbers.
← Previous lesson · All 134 lessons · Next lesson →
The full course — 134 lessons and 402 quiz questions — ships inside the app. Get TPEE to study it offline.
This lesson is the written summary of a course that ships inside the TPEE prompt editor for Windows - thirteen sections, a 0-100 score before you send anything, and it works with no network connection. The three one-time prices, and a 14-day free trial.