Why do iterative prompt-rewrite loops end up with long prompts that generalize worse?

## What collapse looks like A concrete pattern: a loop starts from an 80-word instruction scoring 0.71 on the development set. Ten rounds later the instruction runs to 900 words and scores 0.89 on dev — and 0.63 on a held-out test set it never saw, below where it started. Read the prompt and it is full of clauses like "be very careful", "double-check your answer before responding", and a list of specific cases that look suspiciously like the failures from rounds three through seven. Every individual round looked like an improvement. The artifact is worse than the thing it started from. ## Mechanism 1 — monotone accretion Ask a model to fix failures and it adds. Deletion requires deciding that an existing sentence is unnecessary, which the rewriter has no evidence for — the failing cases say what is missing, never what is redundant. So the instruction grows monotonically. Two costs follow. Later clauses start contradicting earlier ones, and the executing model has to resolve a conflict you never intended it to see. And long instructions dilute attention: the rules that actually mattered are now one paragraph among twenty, which is why a longer prompt can score worse even on the material it was tuned for. A specific sub-case worth naming: clauses that are *inert*. Successive rounds append "be very careful", "think carefully", "double-check" — each intuitively reasonable, none changing behaviour measurably. They survive because nothing tests whether they do anything. An ablation — delete the clause, re-score — usually shows the score is unchanged, and that clause was pure cost. ## Mechanism 2 — overfitting to the shown failures The meta-prompt shows failing cases so the rewriter can diagnose the general defect. What it often does instead is memorize the cases: the rewrite grows a special rule for each one. This is textbook overfitting, with a natural-language hypothesis space instead of weights, and it has the same signature — training score up, generalization down. It is aggravated by small evaluation sets. Prompt optimization is usually run against tens or low hundreds of items because scoring costs model calls, and at that size a handful of memorized rules can move the dev score several points. It is also aggravated by showing the same failures round after round, which lets successive rewrites pile rule upon rule for the same idiosyncratic items. ## Mechanism 3 — optimizing the scorer Any automated score is a proxy for what you actually want. A loop with enough rounds will find the gap. If the objective rewards long, hedged, structured answers, the loop discovers prompts that produce long, hedged, structured answers regardless of whether they are more correct. The result is a prompt that satisfies the measurement and not the goal — Goodhart's law arriving through a rewrite loop. The tell is a score that keeps improving while human spot-checks of the outputs do not. ## How you detect it - **Two-set discipline.** Score every round on a held-out split the rewriter is never shown, and plot both curves. Divergence is collapse. If the two sets track each other, the gains are real. - **Prompt length over rounds.** A curve that only rises is a warning even when scores look fine. - **Per-slice scores.** Aggregate accuracy hides the case where the prompt got better on the majority class and much worse on a minority slice that the shown failures happened not to include. - **Read the diff.** Someone should read what changed each round. Clauses naming specific dev items, or exhortations with no operational content, are visible in seconds and invisible in a metric. - **Ablation.** Delete a suspicious clause and re-score. If nothing moves, the clause was never doing work. ## How you prevent it **Hold out data the rewriter cannot see, and rotate it.** A fixed held-out set consulted every round eventually leaks through your own decisions about when to stop; rotating splits, or reserving a final set touched only once, keeps the estimate honest. **Budget the prompt.** Impose a word or token ceiling in the edit instruction, or require that each round delete at least as much as it adds. Forcing a deletion decision surfaces which clauses are load-bearing. **Keep a champion.** Track the best held-out score seen and keep that version, not the latest. Loops that always ship the last round guarantee that a bad final round becomes production. **Vary the evidence.** Sample different failures each round, cluster them, and prefer showing three distinct failure modes over ten instances of one — the rewrite then has to find the rule rather than enumerate cases. **Stop early.** Halt when held-out improvement is within noise, when the prompt exceeds its budget, or when the critiques go generic. More rounds are not free and, past the turn, are actively harmful. **Keep the human in the diff.** A model-written production prompt should be reviewed, versioned and rollback-able like code. The reviewer's job is precisely the one the metric cannot do: noticing that the instruction now encodes the dev set.