In automatic prompt optimization, how do you pick the scoring metric for a task?

## The scorer is the objective An automatic prompt-engineering loop generates candidate prompts, scores each one on a dataset, keeps the winners and repeats. Whatever function produces that score *is* the definition of "better" for the entire run — the search has no other access to quality. Picking it is therefore the most consequential decision in the setup, and it is driven by the shape of the task's output rather than by preference. ## Families of scorer **Exact match and accuracy.** When each example has one correct answer — a label, a numeric result, a normalized span — you compare the candidate prompt's output to the reference after normalization (case, whitespace, punctuation, formatting). This is cheap, deterministic and un-gameable at the string level. Its weakness is brittleness: a right answer phrased differently scores zero, so normalization rules quietly become part of your metric. **Precision, recall and F1.** When the label distribution is skewed, or when the output is a set of extracted items rather than one value, per-class precision and recall are the honest measurement and F1 combines them into one number the search can rank on. Which of precision or recall matters more is a business decision, not a metric decision, and it is legitimate to optimize a weighted F-beta instead. **Overlap metrics.** ROUGE, BLEU and chrF compare n-grams between the candidate output and a reference text. They are cheap and fully automatic, which makes them tempting as a search objective for summarization or translation, but they measure surface overlap, not usefulness. On meeting-minutes summaries it is common for ROUGE rankings and human preference rankings over the *same* candidate prompts to disagree substantially, because a summary can share many n-grams with the reference while omitting the one decision that mattered. Use them as a coarse screen and know their bias: recall-oriented variants reward length. **Execution-based scoring.** If the output can be run or mechanically verified, run it. Generated SQL is executed against a fixture database and the result set compared; generated code is run against unit tests; generated JSON is validated against a schema; generated data-cleaning regexes are applied to a table of raw-to-expected string pairs and scored by how many pairs they normalize correctly. This is the strongest option when it exists: it is objective, costs no model calls, has no rater bias, and it scores the *behaviour* rather than the wording. It still has a hole — a candidate can special-case the fixtures instead of generalizing — which is why the fixture suite must contain examples the search never sees. **Judge models.** For open-ended text where nothing programmatic applies, an LLM scores the output against a rubric. It is the most flexible and the most expensive scorer, and it introduces the scorer's own errors into the objective. ## Composite objectives Real setups rarely use one number. A common structure is a gate plus a grade: hard constraints (output parses, matches the required schema, stays under a length cap, contains no disallowed content) are pass/fail and zero out a candidate; a graded metric then ranks the survivors. Weighted sums of several metrics are also used, but every weight you add is another dial the search will exploit, so keep the combination small and justify each term. ## Cost and noise are part of the choice The scorer runs once per candidate per example, so it is invoked thousands of times in a campaign. A programmatic scorer costs microseconds and lets you evaluate widely; a judge costs a model call and constrains how many candidates you can afford. Noise matters equally: if the metric is high-variance — a judge at nonzero temperature, or a sampled generation on a small set — the search will happily promote candidates whose apparent lead is sampling noise. Deterministic scoring (fixed decoding where the provider allows it, fixed fixtures) and adequate example counts are how you keep the signal above the noise floor. ## Failure modes to name The metric can be *valid but noisy* (right construct, too few examples), *invalid but stable* (a clean number that measures the wrong thing, like ROUGE for decision-capture), or *gameable* (a candidate raises the score without improving the task). These are three different diseases with three different treatments: more data, a better construct, and adversarial held-out checks respectively. A strong answer distinguishes them instead of saying "use a better metric". ## What interviewers listen for They want you to reach for the most objective scorer the task admits, to be explicit that overlap and judge scores are proxies, to mention hard constraints as gates, and to acknowledge that the metric you optimize hard against will eventually be the metric you get — including its defects.