TPEE Toptronic® Prompt Engineering Editor

133. Distillation 4 - Proving It: What Distillation Actually Transfers

Evaluation designed for a student, not for a leaderboard.

By Jacques Botte, founder of Toptronic®. Last updated 17 October 2026.

The lesson

The only question that matters is comparative: does the student do YOUR task as well as the alternative you would otherwise ship - the teacher via API, or a bigger model, or a prompt - at the cost and latency you need? That sentence defines the evaluation. Same prompts, same judge, same scoring rule, and the teacher included as a baseline rather than as an aspiration.

Measure four things, not one. Task success on a held-out set drawn from traffic the student never saw. Cost per task and latency at the ninetieth percentile, because those are the reasons the student exists. Failure-mode composition - what the student gets wrong when it is wrong, which is usually different from the teacher's failure modes. And behaviour on out-of-distribution input: the empty prompt, the hostile prompt, the other language, the very long document.

Judge design is where distillation evaluations go wrong. If a model judges its own family's output, it prefers it. If a single rubric is applied to a format the student learned and the teacher did not, the student wins on style and loses on substance without anybody noticing. Use at least one deterministic check per task (schema validity, unit tests, exact-match fields) and reserve human or model judgement for what cannot be checked.

Then look specifically for the transfer that did NOT happen. Students commonly inherit the teacher's openings, its hedging vocabulary and its list-shaped answers, while missing the step where the teacher actually reasons. The symptom is plausible and shallow, and it is invisible to a style-based judge. Reading twenty outputs against the teacher's twenty is not optional; it is the cheapest instrument you have.

Record the result as a decision, with the numbers and the failures: 'the student reaches 94 percent of the teacher's success on class A at 12 percent of the cost; it fails on documents over 20,000 tokens, where the teacher succeeds; we route those to the teacher.' That sentence is what makes distillation an engineering choice instead of a bet - and the routing decision in it is usually the real design.

Check yourself

Question 1: What is the correct baseline in a distillation evaluation?
  1. The student's previous checkpoint
  2. The alternative you would otherwise ship - the teacher via API, a bigger model, or a prompt — correct
  3. The public leaderboard
  4. The training loss

Answer: The alternative you would otherwise ship - the teacher via API, a bigger model, or a prompt

The comparison that decides anything is against the thing the student would replace.

Question 2: Why is a judge from the same model family a problem?
  1. It is slower
  2. It tends to prefer its own family's output, and a style-based rubric lets the student win on form while losing on substance — correct
  3. It costs more
  4. It cannot read JSON

Answer: It tends to prefer its own family's output, and a style-based rubric lets the student win on form while losing on substance

Family preference and rubric bias both inflate the student; deterministic checks anchor the evaluation.

Question 3: What is usually the real design decision that comes out of a distillation result?
  1. Choosing a bigger student
  2. Routing: which inputs go to the student and which go to the teacher — correct
  3. Changing the temperature
  4. Shortening the prompts

Answer: Routing: which inputs go to the student and which go to the teacher

A measured boundary between student and teacher, written down, is the engineering outcome.

← Previous lesson · All 134 lessons · Next lesson →

The full course — 134 lessons and 402 quiz questions — ships inside the app. Get TPEE to study it offline.

This lesson is the written summary of a course that ships inside the TPEE prompt editor for Windows - thirteen sections, a 0-100 score before you send anything, and it works with no network connection. The three one-time prices, and a 14-day free trial.