TPEE Toptronic® Prompt Engineering Editor

130. Distillation 1 - A Small Model Learning From a Big One's Outputs

What distillation is, why soft targets carry more than the answer, and what it does not transfer.

By Jacques Botte, founder of Toptronic®. Last updated 17 October 2026.

The lesson

Distillation trains a small model - the student - to reproduce the behaviour of a large one - the teacher - on a chosen distribution of inputs. The student never sees the teacher's weights; it sees the teacher's OUTPUTS, and learns from them. That is the whole idea, and everything difficult about distillation follows from it.

The classic form uses soft targets: instead of 'the answer is class 3', the teacher supplies a probability distribution over all classes, which says how close class 3 was to class 7. A temperature parameter flattens that distribution so the small differences become visible to the student. This is why distillation can beat training on hard labels alone - the teacher is transferring its uncertainty, not just its verdict.

For language models the soft-target form is awkward, because the vocabulary is enormous and many teachers are only reachable through an API that returns text. So the common form is sequence distillation: generate a large corpus of teacher responses to your own prompts, then fine-tune the student on them. It is simpler, it is what most people mean by distillation in 2026, and it loses the uncertainty signal - which is the first thing to know about the trade.

What transfers well: style, format, tone, output structure, the shape of an explanation, domain vocabulary, and narrow task behaviour on inputs that resemble the corpus. What transfers poorly: facts (they can be wrong, and a fact baked into weights cannot be corrected, attributed or removed), depth of multi-step reasoning, and the ability to handle inputs unlike anything in the corpus.

The economic reason it exists is simple: a student that is within a few points of the teacher on YOUR task, at a tenth of the cost and a fifth of the latency, is often the better product. The professional question is never 'can we distil' but 'on which task, measured how, and what does the student do when the input leaves the distribution it was trained on'.

Check yourself

Question 1: What does the student learn from, in distillation?
  1. The teacher's weights
  2. The teacher's outputs on a chosen distribution of inputs — correct
  3. The teacher's licence
  4. The teacher's training data

Answer: The teacher's outputs on a chosen distribution of inputs

The student never sees the teacher's weights - it learns from outputs, which is why the corpus defines the ceiling.

Question 2: What does a soft target carry that a hard label does not?
  1. Speed
  2. The teacher's uncertainty - how close the alternatives were — correct
  3. The licence
  4. The tokenizer

Answer: The teacher's uncertainty - how close the alternatives were

Flattened distributions expose relative confidence, which is the information a hard label throws away.

Question 3: Which of these transfers POORLY?
  1. Output format
  2. Tone and style
  3. Facts and deep multi-step reasoning — correct
  4. Domain vocabulary

Answer: Facts and deep multi-step reasoning

Facts can be wrong and cannot be corrected or attributed once in the weights; reasoning depth is not in the surface of the outputs.

← Previous lesson · All 134 lessons · Next lesson →

The full course — 134 lessons and 402 quiz questions — ships inside the app. Get TPEE to study it offline.

This lesson is the written summary of a course that ships inside the TPEE prompt editor for Windows - thirteen sections, a 0-100 score before you send anything, and it works with no network connection. The three one-time prices, and a 14-day free trial.