TPEE Toptronic® Prompt Engineering Editor

119. Fine-Tuning 1 - When Fine-Tuning Is the Wrong Tool

The honest order of remedies: prompt, then context, then tools, then fine-tuning - and the four cases where tuning is right.

By Jacques Botte, founder of Toptronic®. Last updated 17 October 2026.

The lesson

Fine-tuning is the fourth thing to try, not the first. The order that saves the most money is: write a better prompt, supply the missing context, give the model a tool or a schema, and only then train. Each earlier step is reversible in minutes and costs nothing per iteration; a training run costs hours, a dataset, and a model version you now own and must re-evaluate forever.

Most complaints that sound like a model problem are a context problem. The output is not in the format wanted - that is a schema or a stronger instruction. The model does not know the product - that is retrieval. The model is inconsistent - that is temperature, examples, or a check. The model is too slow or too expensive - that is a smaller model, quantization or caching, not training.

Four cases genuinely call for fine-tuning. STYLE that must be exactly the house voice across thousands of outputs. FORMAT that must be produced reliably without a long instruction on every call. A NARROW TASK where a small tuned model beats a large general one on cost and latency. And BEHAVIOUR that instruction-following cannot reach at all, such as a specialised classification or a domain dialect. Notice that none of them is 'make it know more facts' - facts belong in retrieval, because a fact baked into weights cannot be corrected, attributed or removed.

There is a fifth, less happy case: a model you must run locally, where the largest model that fits is not good enough on your task, so a tuned smaller model is the only path. That is a hardware constraint, and it is legitimate - but it should be named as one rather than presented as an accuracy strategy.

The cheapest fine-tuning that works is often not fine-tuning at all. LoRA and its quantized variant adapt a frozen model with a small number of extra parameters, which cuts the memory and the risk, and can be trained on a single consumer GPU. If the honest answer to 'do we need a fine-tune?' is 'we need a different prompt and one example', that answer is worth more than the run.

Check yourself

Question 1: What is the right order of remedies?
  1. Fine-tune, then prompt, then context
  2. Prompt, then context, then tools or schema, then fine-tuning — correct
  3. Buy a bigger model first
  4. Fine-tune and prompt at the same time

Answer: Prompt, then context, then tools or schema, then fine-tuning

Every step before fine-tuning is reversible in minutes and free per iteration. Training produces an asset you must own and re-evaluate.

Question 2: The model does not know your product's current prices. What is the fix?
  1. Fine-tune on the price list
  2. Retrieval - the prices belong in context, where they can be corrected and attributed — correct
  3. Raise the temperature
  4. Use a bigger model

Answer: Retrieval - the prices belong in context, where they can be corrected and attributed

A fact baked into weights cannot be corrected, attributed or removed. Facts belong in retrieval.

Question 3: Which is a genuine reason to fine-tune?
  1. To add facts
  2. To make a narrow task cheaper and faster with a smaller tuned model — correct
  3. To fix a missing schema
  4. To reduce latency of a hosted API

Answer: To make a narrow task cheaper and faster with a smaller tuned model

Style, format, narrow task economics and behaviour unreachable by instruction are the four. Facts are not one of them.

← Previous lesson · All 134 lessons · Next lesson →

The full course — 134 lessons and 402 quiz questions — ships inside the app. Get TPEE to study it offline.

This lesson is the written summary of a course that ships inside the TPEE prompt editor for Windows - thirteen sections, a 0-100 score before you send anything, and it works with no network connection. The three one-time prices, and a 14-day free trial.