111. Loop Engineering 5 - Budgets, Backoff and Failure Exits

The lesson

A budget has four dimensions and they are not interchangeable. Turns bound how much the loop can try; wall-clock time bounds how long a person waits and how much a lease is held; tokens bound the context cost, which grows faster than the turn count because every turn appends; money bounds what the operator will tolerate. Set all four, and make the smallest one the one that actually fires, because that is the one you have thought hardest about.

Backoff is how a loop asks again without becoming a denial-of-service attack on its own dependency. Double the delay between attempts, add jitter so that a hundred parallel loops do not retry in lockstep, cap the delay at something a human would still call reasonable, and give up after a stated number of attempts. The jitter matters more than the doubling: the synchronised retry storm is the classic cause of an outage that survives the original fault.

Distinguish the three kinds of failure, because they need three different answers. A TRANSIENT failure (a timeout, a 429, a locked file) deserves a retry with backoff. A PERMANENT failure (a missing file, a bad schema, a rejected credential) deserves an exit with a clear message - retrying it wastes the entire budget and produces nothing. An UNKNOWN failure deserves one retry, then a stop and a record, because a loop that keeps trying something nobody understands is how a small fault becomes a long incident.

The exit is the part people skip, and it decides whether the loop is safe. A clean exit releases the lease, writes the state file, records the reason and tells somebody. A dirty exit leaves the lock held and the state half-written, and the next run inherits a mess it cannot interpret. Write the exit first: what is true after a success, what is true after a give-up, and what is true if the process is killed mid-turn - and make the state file atomic so that the third answer is a real one.

Finally, make the failure loud and make it boring. One line to a log file that nobody reads is not a failure signal; a summary that appears where the operator already looks, naming the loop, the exit reason, the budget spent and what is left half-done, is. The measure of a good loop is not how rarely it fails but how quickly a person can tell that it did, and what they have to clean up when it has.

Check yourself