Your product's promptfoo red-team results are unacceptable and you can move three things: the backing model, the application's system prompt, or add a guardrail in front. How do you sequence the experiments over one frozen generated suite, and what do you refuse to change while the campaign runs?

### The tension, and how to resolve it Attribution wants one variable at a time; the deadline wants the biggest fix first. Resolve it by splitting the campaign into **discovery** and **confirmation**. Discovery is cheap and permissive: try several system prompts, look at movement per plugin category, form a hypothesis, change several things at once if it helps you think. Confirmation is strict: one clean single-variable comparison from a stated baseline over the frozen suite. Only the confirmation goes in the report. Conflating the two is how organisations end up with a number nobody can reproduce. ### Ordering, and the costs that drive it ```text change test cost ship / revert cost surface disturbed prompt low low (edit and redeploy) none guardrail medium medium (false-positive blocks before the model risk on real users) model high high (re-baselines every- everything downstream thing tuned before it) ``` **Prompt first.** Cheapest to iterate, cheapest to revert, and its effect is directly visible in the same measurement — no new component, no new counting convention. Each iteration costs one full replay of the suite: for an 800-case suite with multi-turn strategies, a few thousand target calls plus a grader call per case, tens of minutes of wall clock, single-digit dollars against a commodity model and considerably more against a frontier one. Ten prompt iterations is a real bill; budget it before you start, because it is the line item that quietly grows. **Guardrail second.** It moves the measurement surface — blocked requests never reach the model, so those cases now score a classifier rather than your application — and it carries a cost this suite structurally cannot see. Budget a *second* experiment on benign traffic (a few thousand real or realistic ordinary requests, measured for wrongly-blocked rate) before the guardrail's promptfoo delta means anything about production. **Model last.** Most expensive to test, most expensive to ship, and it invalidates work: any prompt you tuned against the old model has to be re-confirmed against the new one, because prompt effects do not transfer cleanly across model families. ### What is frozen for the whole campaign The generated suite file, the grading model and its rubric (pinned explicitly, not inherited from a default that a vendor can update), the target decoding settings, and the convention for counting blocked and errored cases. Change any of those mid-campaign and every earlier run silently becomes incomparable — you have not just lost the next comparison, you have discarded the work already paid for. ### The failure mode that defines this level: overfitting the frozen suite A frozen suite is the price of attribution and the source of its worst failure. Iterate a prompt against the same cases long enough and you are tuning to those cases specifically: the score climbs while nothing generalises. This is not hypothetical — it is the expected outcome of any optimisation loop pointed at a fixed objective. Countermeasures, all three needed. Hold out a stratified slice at the start (a fixed fraction per plugin category) that you never look at during tuning, and confirm the winning configuration there — the gap between tuning-slice gain and held-out gain *is* the overfitting measurement. Refresh the tuning slice on an announced cadence, accepting that a refresh is a new denominator: the trend line breaks and every configuration you still care about must be re-measured on the new suite, which is exactly why refreshes are planned and budgeted rather than incidental. And read the wins: if they concentrate in the exact phrasings the prompt now names, you have memorised the exam, not learned the subject. ### What the suite never tells you An attack-only, generator-authored suite is silent on false positives, latency, cost per request, behaviours no configured plugin produces, and any attack a motivated human would invent that the attacker model did not. A guardrail can top the campaign and still be unshippable; a prompt can win here and lose on task quality, which this suite does not measure either. ### The deliverable A per-variable, per-category effect table on a named suite version, with a named grader, a stated counting convention, the held-out confirmation, the interaction between prompt and guard measured rather than assumed, and an explicit list of what was not tested. That is a decision record, and it survives the question "how do you know?". A single improved percentage does not.