TPEE Toptronic® Prompt Engineering Editor

120. Fine-Tuning 2 - Data, Format and the Training Run

Build a dataset that teaches the thing you meant: examples, splits, hyperparameters and the run you can reproduce.

By Jacques Botte, founder of Toptronic®. Last updated 17 October 2026.

The lesson

A fine-tune learns what is in its dataset, including its mistakes. Five hundred examples of the format you want will teach the format; five hundred examples where a third are subtly wrong will teach that too, and the model will not tell you which third it learned. So the dataset is the project, and the training run is the easy part.

Collect from real work, not from imagination. The best examples are the ones your team already produced and approved: real inputs, real approved outputs, including the awkward ones. Synthetic examples are useful for coverage and dangerous for tone, because a model generating its own training data amplifies its own habits. If you must synthesise, have a person approve every example before it enters the set, and record who approved it.

Split before you look. A held-out set that was chosen after seeing the results is not a test, it is a souvenir. Keep a fixed evaluation split - at least a hundred examples if you have them - that the training never sees, drawn from the same distribution as production, and include the hard cases on purpose. Then the number you report at the end is a number about the future rather than about the past.

The run itself has few decisions that matter and many that distract. Learning rate too high destroys what the model knew; too low changes nothing you can see. Too many epochs memorise the set and lose the general behaviour; one or two is usually enough for a style or format adaption. Batch size is a memory constraint more than a quality one. Record every value you chose, with the dataset hash and the base model revision, so the run can be repeated - an unreproducible fine-tune is an anecdote with a loss curve.

Watch the loss curve for the two shapes that always mean trouble: a validation loss that rises while training loss falls is memorisation, and a validation loss that never moves means the data or the labels taught nothing. And keep a sample of outputs from every checkpoint. The number that matters is not the loss; it is whether the outputs are the ones you would ship.

Check yourself

Question 1: Why is the dataset the project rather than the training run?
  1. Because training is expensive
  2. Because the model learns what is in the dataset, including its mistakes — correct
  3. Because datasets are hard to store
  4. Because the run is not reproducible

Answer: Because the model learns what is in the dataset, including its mistakes

A third of subtly wrong examples teaches the wrongness too, and the model will not flag which third it learned.

Question 2: When must the evaluation split be chosen?
  1. After seeing the results, to pick the best cases
  2. Before training, fixed, from the same distribution as production — correct
  3. From the training set
  4. It does not matter

Answer: Before training, fixed, from the same distribution as production

A held-out set chosen after the results is a souvenir, not a test. Include the hard cases on purpose.

Question 3: Validation loss rises while training loss falls. What is happening?
  1. The model is improving
  2. It is memorising the training set and losing general behaviour — correct
  3. The learning rate is too low
  4. The dataset is too large

Answer: It is memorising the training set and losing general behaviour

The other bad shape is a validation loss that never moves, which means the data or labels taught nothing.

← Previous lesson · All 134 lessons · Next lesson →

The full course — 134 lessons and 402 quiz questions — ships inside the app. Get TPEE to study it offline.

This lesson is the written summary of a course that ships inside the TPEE prompt editor for Windows - thirteen sections, a 0-100 score before you send anything, and it works with no network connection. The three one-time prices, and a 14-day free trial.