After a model upgrade, half your frozen jailbreak regression entries stop firing — what do you report the suite is worth?

## What the question is really asking This is a judgment call under organisational constraint, not a technical one. The suite exists, it cost something, other teams cite it, and after an upgrade half of it has gone quiet. Somebody has to say what it is worth now, in a form a budget owner can act on — and has to be willing to say "less than you think" about an asset they themselves built. ## The three buckets in the report **Still testing something.** Entries that record the *family* they instantiate, not merely a string. These can be rebuilt for the new model's turn conventions and rerun, and the before-and-after rates mean something because you can re-fit the wording each time. **Decoration.** Entries that are a frozen wording and nothing else. Once the boundary moves they run, pass, and measure nothing. These are worse than absent, because a green line in a report is read as coverage by people who will not read the caveat. **Unattributable.** Entries whose trial counts were too small to distinguish a real change from noise. They should be reported as unknown, not as passes. ## The recurring cost you are pricing A suite cut around families is not a one-off rewrite. Each family needs several fresh instantiations, each needs repeated trials to get a rate rather than a bit, and this repeats per route and per release. Behind an internal gateway that repoints routes without telling the app teams, releases arrive silently, so the work is triggered by something nobody is watching. That is the structural problem: the decay is continuous and invisible, while the budget conversation is annual. The honest options are usually: | Option | What you get | What it costs | | --- | --- | --- | | Small family-level suite, re-instantiated each release | A rate per family over time, comparable across upgrades | Recurring effort per family, per route, per release | | Point-in-time exercises, no standing suite | Depth on whatever is current | No trend, no before-and-after, nothing to cite between exercises | | Keep the frozen library unchanged | Nothing, at low apparent cost | Steadily growing false confidence | The third is the default outcome when nobody funds the second, and naming it as an outcome rather than a status quo is much of the value of the report. ## Where the cost lands, and who can refuse The team that owns the gateway swaps models; the team that owns the suite absorbs the re-fitting. That mismatch is the reason these suites rot: the party creating the work is not the party paying for it. A lead can legitimately refuse the whole thing — declare that a standing suite is not worth maintaining at this cadence, retire it explicitly rather than let it idle green, and buy periodic family-level work instead. What is not legitimate is keeping it and letting others cite it. ## The claims to refuse, in writing - "The upgrade fixed these weaknesses." A non-firing frozen entry cannot support that; the wording may simply no longer land. - "The suite is green, so the models behind the gateway are safe." It shows that recorded strings did not land on the routes tested on that day. - "We tested the new model." You replayed instantiations fitted to the old one. ## What durable investment looks like here The asset with a long half-life is the catalogue of families and the measurement discipline around them — what mechanism, how many instantiations, what rate, on which route, on what date. The library of strings has the short half-life, and it shortens further every time a vendor ships. Saying that plainly, and attaching a price to the alternative, is the deliverable a lead is expected to produce here.