A prompt search shows no gain for six rounds — how do you decide it has converged?

## Why "the score stopped moving" is not enough Every score in an automatic prompt-engineering loop is an estimate. It carries variance from decoding randomness, from judge or scorer variability, and above all from the finite evaluation set. With 200 items and accuracy near 0.7, the standard error of a single measurement is roughly 3 percentage points, so two prompts whose true quality is identical will routinely appear 4 or 5 points apart, and a genuinely better prompt can look flat for a round or two. Declaring convergence from a flat best-score curve therefore conflates two very different situations: the search has genuinely run out of reachable improvements, or it is producing improvements too small to see through the noise floor. ## Step one: measure the difference, not the scores The cheapest large win is **paired evaluation**. Score the incumbent and every challenger on exactly the same items, then compare per-item outcomes rather than aggregate percentages. Item difficulty cancels out of the comparison, so the variance you are fighting drops sharply and a much smaller evaluation set gives a reliable ranking. From the paired outcomes you can compute a simple interval — a bootstrap over items, or a sign test over items where the two prompts disagree — and answer the actual question: is the best challenger's advantage distinguishable from zero at this sample size? If the interval straddles zero, you do not have evidence of a plateau; you have evidence that your evaluation resolution is coarser than the improvements on offer. The options are to increase items per comparison for the top candidates only, or to accept that further gains are below the resolution you are willing to pay for and stop deliberately. ## Step two: classify the plateau If the flatness is real, diagnose which kind it is. **Local optimum from the operator set.** The neighbourhood you defined cannot reach anything better. This is the common case for greedy search with conservative operators — every proposal reweighs the same phrasing. The tell is that all recent proposals are minor variants of the incumbent. Remedies: a random restart from a different seed prompt, a deliberately large perturbation of the incumbent, a broader operator set that can restructure sections rather than reword them, or carrying a population instead of a single incumbent so several regions stay alive. **Task or data ceiling.** The remaining failures are not addressable by prompting. Read fifty failing items by hand and bucket them. If a large share are mislabelled gold answers, ambiguous cases where reasonable humans disagree, or items that require context the prompt was never given, no rewording will fix them and the measured ceiling is a property of the dataset, not the prompt. This diagnosis is worth more than another thousand model calls, and it redirects effort to data curation, retrieval, or a different model. **Scorer saturation.** The metric may already be near its maximum on the items the loop sees, so improvements land in a region the score cannot express. That again ends the useful life of this loop rather than justifying more rounds. ## Step three: encode the decision as a stopping rule Do not decide plateau by eye in production runs. Standard practice combines several rules and stops on whichever fires first: - **Patience**: stop after N consecutive rounds without an improvement exceeding a threshold (a threshold larger than the noise floor you measured, not zero). - **Hard budget cap**: a maximum number of rounds, model calls or dollars, set before the run. - **Marginal-value rule**: stop when the improvement per unit of spend falls below what the improvement is worth to you. - **Restart policy**: rather than stopping outright, spend the remaining budget on a fresh restart; several independent short runs frequently beat one long one on a rugged landscape, and they also give you a spread of outcomes rather than a single point. ## Step four: confirm the winner honestly The best dev score at the end of a long search is an optimistic number, because the winner was chosen by that same set. Confirm the chosen prompt on data the loop never scored against, once, and report that number. If the confirmation score is much lower than the search score, the plateau debate was moot: the recent rounds were fitting the evaluation set rather than improving the prompt. ## What a strong answer sounds like Name the noise floor and how you measured it, describe paired comparison as the first move, distinguish local optimum from ceiling using failure inspection, state the concrete stopping rule you would configure, and say what you would do with the remaining budget — restart, widen operators, or stop and reinvest in data.