84. Make the Model Its Own Critic: Self-Refinement and Verbal Reinforcement

The lesson

Large language models rarely produce their best work on the first try, yet most users send one prompt and accept one answer. In 2023, two research papers showed that a model can become its own best editor: Self-Refine ("Self-Refine: Iterative Refinement with Self-Feedback" by Aman Madaan and colleagues) and Reflexion ("Reflexion: Language Agents with Verbal Reinforcement Learning" by Noah Shinn and colleagues). Both papers come from the TPEE library of scientific papers, which you can also browse on tpee.app/papers.

Self-Refine's core idea is a loop in which the SAME model plays three roles: generator, feedback provider, and refiner. The model produces a draft, then critiques it, then rewrites it in light of that critique - and the loop repeats until it stops improving. No additional training, fine-tuning, or reinforcement learning is required; everything happens in the prompt and context layer.

The evidence for Self-Refine is strong: across 7 diverse tasks (from dialog response generation to mathematical reasoning) and three models (GPT-3.5, ChatGPT, and GPT-4), iterative self-refinement improved task performance by about 20% absolute on average, and even state-of-the-art GPT-4 got better at test time. The first draft is not the ceiling.

The critique prompt is the whole game. A vague "review your prompt and make it better" produces vague, useless feedback. Effective Self-Refine-style critique is specific and actionable: which requirement is unmet? Which section is unclear? What exactly should change next? Writing a good rubric for the critic is a prompt-engineering skill in its own right - and one you control completely.

Reflexion extends self-critique across ATTEMPTS. Instead of refining one output, a Reflexion agent verbally reflects on a failure - writing what went wrong and what to change - and stores that reflection in an episodic memory buffer that shapes the next attempt. This is reinforcement learning carried out in language: no weight updates at all.

The evidence for Reflexion is striking: 91% pass@1 on the HumanEval coding benchmark, surpassing the previous state of the art - GPT-4 at 80% - achieved purely by prompting an agent to reflect on its failures and remember them. Verbal feedback replaced gradient updates.

Practical patterns you can use today: (a) run two-pass prompts - ask for the draft in one message, then ask for a critique against a checklist before any rewrite; (b) ask for the critique BEFORE asking for the revision, so the model judges rather than defends; (c) keep a "lessons learned" block in your project prompts - TPEE's version history is your episodic memory buffer, and the Examples and Data sections are natural places to store reflection text; (d) when an agent fails, demand a written post-mortem, not just a retry.

Know the limits. Self-critique is not ground truth: models can be confidently wrong about their own output, feedback quality drops on tasks the model is weak at, and every loop iteration costs tokens. Pair self-reflection with external checks - tests, facts, TPEE's prompt-scoring rubric. The takeaway: the cheapest quality gain is often a second, critical pass, and TPEE's 12-section editor with its score panel is a built-in place to run that loop.

Check yourself