TPEE Toptronic® Prompt Engineering Editor

113. Loop Engineering 7 - Operating Loops in Production

Run loops as systems: observability, cost control, change management, and the review that decides whether a loop should exist.

By Jacques Botte, founder of Toptronic®. Last updated 17 October 2026.

The lesson

A loop in production is a service, and it is judged like one. What is its success rate per hundred runs, what does one run cost, how long does it take at the ninetieth percentile, and how often does it exit by giving up rather than by finishing? Those four numbers, recorded per run, are the difference between operating a loop and owning a rumour about one.

Observability for a loop has three layers and most teams build only the first. The first is the transcript: what the model saw and did. The second is the DECISION record: the goal, the check's verdict at each turn, the budget remaining when it stopped. The third is the OUTCOME: what changed in the world, and whether a person would call that change correct. A loop with only a transcript cannot answer the only question that matters after a bad run - why did it stop there?

Cost control is a design constraint, not a finance report. Cap the spend per run and per day, alert at a stated fraction rather than at the limit, and treat a sudden change in average turns as a defect signal, because it usually means the check started failing for a new reason. The cheapest loop is the one whose check is precise enough that it stops on the first turn that worked.

Loops drift the way any automation drifts: the dependency ships a new version, a test becomes flaky, a vendor changes a default, a prompt stops meaning what it meant. So a loop needs a change-management story - the prompt, the checks and the budget are versioned together, and a change to any of them is a change to the loop - plus a scheduled review, because a loop that nobody reviews slowly becomes a loop that optimises for its own check.

The review question is the one this course keeps returning to. Is this loop doing something a person would still choose if they watched every run? If the answer is no, the fix is not a better prompt; it is a narrower scope, a stronger check, or a person in the loop at the one decision that actually needs judgement. Loops are wonderful at volume, consistency and patience. They are not a substitute for knowing what you meant.

Check yourself

Question 1: Which four numbers should be recorded per run?
  1. Tokens, temperature, model, provider
  2. Success rate, cost per run, p90 duration, give-up rate — correct
  3. Lines of code, files, commits, reviewers
  4. Uptime, latency, throughput, errors

Answer: Success rate, cost per run, p90 duration, give-up rate

Those four describe whether the loop is worth operating. They are the numbers that turn ownership into measurement.

Question 2: What does the decision record add that the transcript cannot?
  1. It is shorter
  2. It says why the loop stopped - the goal, each verdict and the budget remaining — correct
  3. It contains the model's reasoning
  4. It replaces the outcome record

Answer: It says why the loop stopped - the goal, each verdict and the budget remaining

A transcript shows what happened. The decision record answers the question after a bad run: why did it stop there?

Question 3: A loop's average number of turns suddenly rises. What does that usually mean?
  1. The model got smarter
  2. The check started failing for a new reason - treat it as a defect signal — correct
  3. Nothing; turns vary
  4. The budget should be removed

Answer: The check started failing for a new reason - treat it as a defect signal

A stable loop has a stable turn distribution. A jump usually means the verifier changed behaviour, not that the work changed.

← Previous lesson · All 134 lessons · Next lesson →

The full course — 134 lessons and 402 quiz questions — ships inside the app. Get TPEE to study it offline.

This lesson is the written summary of a course that ships inside the TPEE prompt editor for Windows - thirteen sections, a 0-100 score before you send anything, and it works with no network connection. The three one-time prices, and a 14-day free trial.