TPEE Toptronic® Prompt Engineering Editor

132. Distillation 3 - The Run: Temperature, LoRA and the Curve Shapes That Warn You

Training mechanics a practitioner actually tunes, and the two loss curves that mean stop.

By Jacques Botte, founder of Toptronic®. Last updated 17 October 2026.

The lesson

If you have the teacher's logits, use a temperature above 1 - typically 2 to 5 - and mix the soft loss with a hard loss against real labels where you have them. The mix matters: pure soft targets can make the student imitate the teacher's mistakes with great confidence, and pure hard labels throw away the uncertainty you paid for. Record the temperature and the mixing weight; they are hyperparameters of the artifact, not of the run.

If you have text only, this becomes supervised fine-tuning on teacher outputs. Sequence-level cross-entropy, the usual learning-rate schedule, and a decision about masking: train on the response tokens only, not the prompt, unless you specifically want the student to learn to produce prompts. The prompt tokens are conditioning, and paying loss on them dilutes the signal you care about.

For most teams the student is a LoRA or QLoRA adapter on an open base model, not a full fine-tune. That is cheaper, it keeps the base model intact and swappable, and it makes the artifact small enough to version honestly. The trade is capacity: a rank that is too low cannot represent the behaviour, and the symptom is a model that learns the format and not the judgement.

Now the two curve shapes worth memorising. The first is the curve that falls smoothly and then flattens while training loss keeps dropping - that is memorisation beginning, and the validation set is already telling you to stop. The second is a validation loss that falls, rises, falls again in a sawtooth; that is usually a learning rate too high for the batch size, not a data problem, and lowering it is a five-minute fix that saves a day.

Two more signals that are not on the loss chart. Watch the length distribution of the student's outputs against the teacher's: a student that suddenly writes much longer or much shorter answers has learned a formatting artefact, not the task. And watch a fixed sample of twenty prompts by eye at every checkpoint - the eye catches register collapse, repeated openings and confident nonsense long before a benchmark moves.

Check yourself

Question 1: Why mix soft targets with hard labels where you have them?
  1. To halve the cost
  2. Because pure soft targets can make the student imitate the teacher's mistakes confidently, and pure hard labels discard the uncertainty — correct
  3. Because the licence requires it
  4. Because it reduces the context window

Answer: Because pure soft targets can make the student imitate the teacher's mistakes confidently, and pure hard labels discard the uncertainty

The mixture keeps the transfer of uncertainty while anchoring the student to ground truth.

Question 2: Validation loss falls, rises, falls again in a sawtooth. Most likely cause?
  1. Bad data
  2. A learning rate too high for the batch size — correct
  3. A missing licence
  4. Too many layers

Answer: A learning rate too high for the batch size

The instability is an optimisation symptom, and lowering the learning rate is the cheap first fix.

Question 3: Training loss keeps falling while validation flattens. What is happening?
  1. The model is improving
  2. Memorisation is beginning - the validation set is already asking you to stop — correct
  3. The tokenizer is broken
  4. The corpus is too large

Answer: Memorisation is beginning - the validation set is already asking you to stop

A widening gap with a flat validation curve is the classic overfitting signal.

← Previous lesson · All 134 lessons · Next lesson →

The full course — 134 lessons and 402 quiz questions — ships inside the app. Get TPEE to study it offline.

This lesson is the written summary of a course that ships inside the TPEE prompt editor for Windows - thirteen sections, a 0-100 score before you send anything, and it works with no network connection. The three one-time prices, and a 14-day free trial.