Automatic Prompt Engineering

Three sources dominate: inducing an instruction from labelled input/output pairs, resampling paraphrases of a seed instruction, and mutating slots in a fixed template. They trade diversity against staying on-task, and most systems combine them.

A prompt is discrete text, not a continuous parameter vector, so no gradient points toward a better prompt. Optimization instead runs as black-box search: propose edited candidates, score each on a dataset, keep the winners, repeat under a fixed budget.

A DSPy signature declares one step's named inputs and outputs plus a short description of the task — for example context, question -> answer. The wording, formatting rules and examples that make up the actual prompt are generated by the framework, not written by you.

Match the scorer to the output shape: exact match or F1 when one label is right, execution-based checks when the output can be run and verified, overlap scores only as a rough proxy, and a judge model only for genuinely open-ended text.

Each round only ever adds: caveats accumulate because nothing is deleted, and the additions encode the specific failures the rewriter was shown rather than the rule behind them. The prompt fits the development set, so held-out performance falls while the loop reports progress.

Selecting the best of hundreds of candidates on one small dataset captures that dataset's noise along with real quality. The winner's score is the maximum of many noisy estimates, so it is biased upward — and the part that came from noise does not transfer.

Six flat rounds is a plateau only if the flatness survives measurement. Re-score the incumbent and its challengers on the same evaluation items, look at the paired difference and its uncertainty, and only then decide between stopping, restarting from a new seed, or widening the operators.

Compiling returns a new copy of the program whose predictors now carry selected demonstrations (and, with instruction optimizers, rewritten instruction text). No model weights change. That prompt state is the artifact, and it can be saved to JSON and loaded later.

Only for screening. A cheap subsample can rank candidates roughly and cull the obviously bad ones, but promotion decisions and reported numbers must come from the full gold set, because a moderately correlated proxy reorders candidates exactly where they are closest.

Accuracy is dominated by the majority class. On a manufacturing set that is 2% defective, a candidate that never flags a defect still scores 98%, so the optimizer ranks it top while it fails completely on the class you built the system for.

Run the current prompt over training inputs, keep only the traces whose final answer matched the label, and bundle sampled subsets of those verified traces as candidate exemplar sets. The exemplar set is then searched alongside the instruction text.

APE treats writing an instruction as search. An LLM infers candidate instructions from input-output examples of the task, each candidate is scored by actually running it on held-out data, and the best scorers are resampled into semantically similar variants until scores stop improving.

Splitting diagnosis from editing forces the model to state, in writing, why the prompt failed. That written reason constrains the next edit to the observed defect, makes the change reviewable by a human, and gives a stopping signal when critiques go generic.

Mutation edits one parent prompt — rewording an instruction, swapping an exemplar, changing the output format. Crossover splices sections from two parent prompts into a child. Selection then keeps the highest scorers as the next generation's parents.

Swapping the module keeps the same signature but changes how that step is executed: ChainOfThought extends the declared outputs with a reasoning field the model fills before answering. Calling code, training data and metric are untouched — you change one line and recompile.

First confirm the pool is the problem: if candidate scores cluster within noise, the search had nothing to choose between. Then dedup near-identical candidates, generate from multiple seeds, demonstration subsets and proposers, and spend the freed scoring budget on genuinely distinct prompts.

Compare the optimized score against a held-out set the search never touched, and read the top candidates' actual outputs. A score that climbs on the search set while flat or falling on held-out data is the signature of a candidate exploiting the scorer.

Spend unevenly. Score every candidate cheaply on a small shared subset, eliminate the clearly weak ones, and reallocate the saved evaluations to survivors — successive halving. Budget in model calls and tokens, not candidate counts, and reserve capacity to confirm finalists.

Treat the compiled artifact as a build output pinned to the model it was compiled against. On an upgrade, first evaluate the existing artifact on the new model against a held-out set, then recompile and compare all three options — old artifact, new artifact, and uncompiled — before deciding what to ship.

Treat the judge as expensive infrastructure: cascade cheap programmatic checks in front of it so it only ranks survivors, pin its model version and rubric for the whole campaign so scores stay comparable across generations, and re-validate it against human labels on a fixed anchor set.

Generation is usually the cheap half: proposing N prompts costs N calls, while scoring them costs N times the evaluation-set size. That asymmetry argues for a strong proposer and a small, well-chosen pool, with the optimized prompt always validated on the model that will actually serve it.