After searching twenty exemplar orders on a held-out dev set, is the winning order worth shipping?

## Why the winner looks better than it is Searching orders is model selection, and every model selection over a finite validation sample carries a winner's curse. If twenty permutations all have similar true accuracy, the one that scores highest on a dev set is disproportionately likely to be the one that got lucky on that particular sample. The bias grows with the number of candidates and shrinks with dev-set size. On a 200-item dev set, the binomial standard error around a 75% score is about three points, so the spread between the best and worst of twenty near-identical candidates can be six points of pure noise. Reporting the winner's dev score as its expected production accuracy is therefore a systematic overstatement. The corrective is standard and cheap: - **Three splits, not two.** Search on dev, then measure the chosen order once on a test split that the search never touched. That number is the honest one. - **Quantify the noise floor.** Compute a confidence interval for a single order's score at your dev-set size. If the winner's margin over the median is inside it, you have selected noise. - **Repeat the search.** Re-run the same search on a different dev sample. If a different order wins, the effect you are exploiting is not stable. - **Prefer robust orders.** An order that is second-best but sits in a cluster of similar-scoring neighbours is often a safer ship than a lone spike. ## The generalisation question Even a genuine, test-confirmed advantage is tied to the configuration it was found under. The order interacts with the model's weights, the shot count, the template and the label wording. That means: - **Across model versions.** A provider upgrade can reshuffle the ranking. The tuned order does not become bad, but it stops being reliably best, and nothing in your pipeline notices. - **Across distributions.** If the production input mix drifts away from the dev sample, an order tuned to that sample's difficulty profile loses its edge. - **Across shot-set edits.** Adding, removing or replacing a demonstration invalidates the search entirely; the space of orders has changed. So the real cost of shipping a tuned order is not the search — it is the standing obligation to re-run and re-validate it as part of every model migration, plus the evaluation infrastructure that makes that cheap. Teams that skip that obligation ship a prompt whose one measured advantage silently expires. ## Label-free alternatives to the search itself When labelled dev data is scarce, orders can be ranked without labels by looking at the *distribution* of predictions each order induces. The published entropy-based approach generates a probing set from the demonstrations themselves, runs each candidate order over it, and prefers orders whose predicted-label distribution is well spread rather than collapsed onto one label — a degenerate order that answers `approve` to everything is detectable without knowing the right answers. This does not need a labelled set, but it optimises a proxy, so it should still be sanity-checked against whatever labelled data exists. ## The strategic alternative: reduce the sensitivity A principal-level answer weighs tuning against removing the need to tune: | Lever | What it buys | What it costs | |---|---|---| | Ship one searched order | Free at inference time | Search plus revalidation on every change; fragile | | More demonstrations | Lower permutation variance | Tokens, latency, context budget | | Label-balanced shot block | Removes the majority-label component | Requires enough exemplars per class | | Prior correction on outputs | Removes most systematic skew | Needs visible label probabilities; closed label set | | Vote across k orders | Variance averaged away; no tuning to maintain | k× tokens and latency | | Fine-tune the behaviour | Prompt-permutation surface disappears | Training and refresh pipeline, data, eval discipline | The decision is workload-shaped. For a low-volume internal tool, one searched order plus a note to revalidate is proportionate. For a high-volume decision system with an expensive minority class, spending tokens on ensembling or engineering effort on fine-tuning buys stability that a permutation search never can. For anything in between, the honest position is that the sensitivity is a measured property of the system with a known magnitude, and the choice is which cost you would rather carry. ## What an interviewer is listening for Three things. First, that you know best-of-k on a small sample is a biased estimate and you name the fix (an untouched test split, a confidence interval, a repeated search). Second, that you treat transfer across models and distributions as an open question to test, not an assumption. Third, that you can argue for *not* shipping the tuned order — that the variance is a system property worth engineering down rather than a lottery worth playing. Candidates who describe the search but never mention validating the winner have optimised a number rather than a system.