Why does a prompt-rewrite loop critique the prompt before editing it?

## The loop shape The naive self-refinement loop is one call: current prompt plus failures in, new prompt out. The critique-then-edit variant splits that into two calls with different jobs. 1. **Critique.** "Here is the instruction, here are cases where it produced the wrong output. Explain why the instruction allowed those outputs." The output is prose — a diagnosis, not a prompt. 2. **Edit.** "Here is the instruction and this diagnosis. Rewrite the instruction to address exactly this, changing nothing else. Output only the new instruction." The analogy that gave the pattern its name is optimization: the critique plays the role of a gradient — a written statement of which direction to move — and the edit plays the role of the step. The analogy is loose (there is no metric being differentiated and no guarantee of descent), but it captures the useful part: separating *where to go* from *taking the step*. ## Why the extra call earns its cost **Targeted edits.** Asked to rewrite directly, models tend to produce a fresh instruction rather than a modified one — which throws away phrasing that was working, and makes each round's diff unreadable. Conditioning the edit on a specific diagnosis keeps the change local and the diff small enough to review. **Inspectability.** The diagnosis is the artifact a human can actually audit. "The instruction says 'classify the incident' but never lists the allowed labels, so the model answers in free text" is a claim you can agree or disagree with in seconds. The rewritten prompt alone gives you nothing to check except by re-reading it in full. **Reusable signal.** Diagnoses across rounds aggregate into a picture of how the prompt is failing — ambiguity in one clause, missing output format, an unstated tie-breaking rule. That picture often tells you the prompt was never the problem: if the diagnosis keeps saying "the model lacks the policy text", the fix is retrieval, not phrasing. **A stopping signal.** This is the underrated one. As long as critiques name specific defects, the loop has signal. When they degrade into content-free exhortation — "the instruction should emphasise care", "the model should double-check its answer" — the critic has nothing left to diagnose and every further round will append inert clauses. That degeneration is easier to spot in the critique stream than in the prompt itself, where each addition looks locally reasonable. ## Grounding the critique A critique of a prompt read in isolation is speculation: the model will find things to say about any instruction. The critique must be conditioned on real failing outputs, ideally with the expected output alongside, so the diagnosis explains an observed discrepancy rather than an imagined one. Include a passing case or two as a control; a critique that condemns behaviour that is actually correct is a signal that the critic is confabulating. ## Separating critic from editor Using one model for both roles is common and works, but the critic inherits the editor's blind spots. Some setups use a stronger model as critic, or run the critique at a lower temperature than the edit. Either way, the critique is an unverified model output. It is a hypothesis about why the prompt failed, and the only thing that confirms it is that the edited prompt scores better on data the critic never saw. ## Failure modes - **Critique that flatters.** Ask a model whether a prompt is good and it will often say yes. Ask *why these specific outputs were wrong* and it has something concrete to work with. - **Diagnosis that names the output, not the cause.** "The model gave the wrong label" restates the failure; "the instruction never says what to do when two labels both apply" names a cause you can edit. - **Edit that exceeds its brief.** If the editor rewrites everything anyway, add an explicit constraint ("change at most two sentences") or diff the result and reject oversized edits. - **Trusting the critique as evidence.** A persuasive diagnosis followed by a prompt that scores worse is common. Score, then decide. - **Running the loop with no cap.** Even with good critiques, each round costs two calls plus an evaluation; set a round cap and keep the best-scoring version rather than the last one. ## Practical shape A workable configuration is three to five rounds, each round: score the current prompt, critique against the worst failures, edit under a size constraint, re-score on a held-out split, keep the champion. Stop early when the critique goes generic or when the held-out score stops moving beyond noise. Keep every round's critique — when someone later asks why the production prompt contains a strange clause, the diagnosis that produced it is the answer.