Continuous automated LLM red-team suites make successful harmful outputs pile up as an archive that doubles as the regression set proving fixes hold. How do you decide what is kept, for how long, and who may open it?

The tension is genuine and does not dissolve: destroy everything and you cannot prove a fix held; keep everything and you have built a growing, indexed, validated attack corpus against your own products, sitting on your own network with internal access. ## Decompose the artefact first A stored run has four separable parts: the **attack input**, the **model output**, the **verdict**, and the **run metadata**. A regression run consumes the input and the metadata, sends the input to the current build, and produces a *fresh* output and a *fresh* verdict. Replaying a stored output would test nothing at all — it would assert that a string you already have is still the string you have. That single observation does most of the work. The default becomes: keep input, metadata and verdict; keep a hash of the completion; and keep the completion itself only where the harm claim rests on properties of that specific text and cannot be re-derived. Those exceptions are real but narrow — grader calibration over time, and defending a contested severity months later — and they belong in the tightest access tier. ## Retention by harm class, not by uniform age Some classes never enter the archive. Others carry a short clock. Regression-critical inputs may justify a longer one, but should still expire by default with renewal as an explicit, signed act by a named person. Indefinite retention is never a decision anybody made; it is the state every archive drifts into when expiry is opt-in. ## Access, ownership, and the archive as a target Access should be by name, with two-person approval, every read logged and the log reviewed — not a group membership granted at onboarding, which turns the corpus into ambient reading for a population that only grows. Read keys should not be held by the CI identity that writes the archive; write and read are different privileges here, and a compromised pipeline should not yield the whole corpus. Ownership belongs to a named accountable person outside the red team, typically in legal or trust-and-safety: the team that benefits from keeping evidence should not be the team that authorises keeping it, and the red team's job is to classify findings into the schedule rather than to set it. Then threat-model the archive itself and put it in incident-response planning, answering plainly: if this store is breached, what does the adversary now have? The answer is a curated, validated playbook of attacks that work against your live products — a higher-value target than most of what the security team is defending. Finally, the humans who review this material need rotation, consent and support; concentrating it on one reviewer is both a welfare failure and a single point of judgement on every severity call. ## What it costs — and this is where the policy actually breaks Storage is trivially cheap, which is why nobody argues about storage. The real cost is regeneration. If the regression suite holds two thousand attack inputs and each is sampled three times to cope with sampling variance, one full run is six thousand model calls; nightly, that is on the order of a hundred and eighty thousand calls a month, plus the grader calls on top if scoring is itself a model. Against a metered frontier endpoint that is a line item somebody will notice, and the tempting saving — cache the completions and re-score the cache — is exactly the move that destroys the point of the run. The honest response is to shrink or tier the suite (a small nightly set, the full set weekly), not to stop regenerating. There is also human cost: access review, the owner's time, and expiry automation that somebody has to maintain. ## Where the number misleads A green regression run is read as "the model is fixed". It supports something much narrower: *these archived inputs no longer succeed against this build*. It says nothing about the attack space outside the archive, and — the sharp edge — a fix narrowly targeted at the archived strings, whether a guard rule from your own team or a provider-side patch, produces exactly the same green result as a genuine fix. An archive frozen at the moment findings were logged is a set the defence can overfit to, and the longer it is used as the pass/fail gate the more the system is tuned to it. Treat archive pass rate as a regression alarm only, never as a coverage measure, and keep a rotating set of fresh, unarchived variants of the same attack family as the control that tells you which kind of fix you got. ## How I would know the policy is more than paper Sample the store against the schedule and confirm items past expiry are actually gone. Read the access log for named, justified reads rather than routine browsing. And verify that a regression run genuinely regenerates output from inputs — because if it has quietly come to depend on stored completions, the entire argument for not retaining them has already collapsed and nobody noticed.