121. Fine-Tuning 3 - Evaluation, Drift and the Cost of Ownership
After the run: prove it beat the alternative, then keep proving it as the world moves.
By Jacques Botte, founder of Toptronic®. Last updated 17 October 2026.
The lesson
A fine-tune must be compared against the thing it replaces, on the same tasks, with the same judge - and the comparison is not just quality. Record success rate, cost per task, latency at the ninetieth percentile, and the failure modes, because a tuned model usually wins on some of those and loses on others. A model that is 3 percent better and 40 percent more expensive to operate is a decision, and it should be made with the number in hand.
The judge matters more than usual here, because you trained on data that resembles your test. Use a held-out set drawn after training from production traffic, and where possible a different judge from the one used during development. If the tuned model only beats the base model under the tuned model's own eye, it has learned the judge, not the task - which is the same defect this course names in every loop.
Drift arrives from three directions. The INPUTS change, as customers ask new things. The WORLD changes, so a fact the model learned becomes wrong - and this is the one reason never to train facts into weights. And the BASE MODEL changes, because the vendor ships a new version and your adapter may not apply to it, or may apply and behave differently. Every one of those needs a re-run of the evaluation set, on a schedule, not when someone complains.
The cost of ownership is the part nobody budgets. There is an artifact to store and version, a base-model revision to pin, a serving path that must load the adapter, an evaluation set that must be maintained, and a person who understands why each hyperparameter was chosen. A fine-tune is not an event; it is a small asset with a maintenance bill, and if that bill is not accepted explicitly, the asset quietly rots until it is worse than the prompt it replaced.
The honest summary: fine-tuning is the right tool for style, format, narrow economics and behaviour instruction cannot reach. It is the wrong tool for facts, for speed of iteration, and for anything you have not measured. And the professional habit is the one this whole course has been building - make the claim, then measure it, then write down what the measurement said, so the next person inherits evidence instead of confidence.
Check yourself
Question 1: Which four things must a fine-tune's comparison record?
- Loss, epochs, batch size, learning rate
- Success rate, cost per task, p90 latency, failure modes — correct
- Model size, provider, licence, price
- Dataset size, split, hash, date
Answer: Success rate, cost per task, p90 latency, failure modes
The run metrics describe training. Those four describe whether the thing you would ship is better.
Question 2: Why must facts never be trained into weights?
- Because it is expensive
- Because a fact baked into weights cannot be corrected, attributed or removed when the world changes — correct
- Because weights are too small
- Because retrieval is slower
Answer: Because a fact baked into weights cannot be corrected, attributed or removed when the world changes
Drift is guaranteed. A wrong fact in a prompt is a one-line fix; a wrong fact in weights is a retrain.
Question 3: A tuned model beats the base model only under its own judge. What has it learned?
- The task
- The judge — correct
- Nothing
- The format
Answer: The judge
Use a held-out set drawn after training and, where possible, a different judge - otherwise you have measured the judge's taste.
← Previous lesson · All 134 lessons · Next lesson →
The full course — 134 lessons and 402 quiz questions — ships inside the app. Get TPEE to study it offline.
This lesson is the written summary of a course that ships inside the TPEE prompt editor for Windows - thirteen sections, a 0-100 score before you send anything, and it works with no network connection. The three one-time prices, and a 14-day free trial.