How do you keep an AI red-team report from being reused months later as a standing safety certificate for a system that has since changed — a swapped model, an edited system prompt, a new tool integration — and what goes into the deliverable to make that explicit?

The failure mode here is procurement, not testing. A competent report from March gets attached to a security questionnaire in November, for a system whose system prompt was edited in April and whose model was swapped in June. Nobody lied at any point. The deliverable simply had no expiry, and nothing in it made the change detectable to the person quoting it. **Fingerprint what you actually drove** An appendix that pins the identifiers determining behaviour, so that every conclusion can be phrased against it rather than against “the system”: ```text APPENDIX A — TESTED CONFIGURATION model provider id + serving tier + any version string exposed sampling temperature / top-p, if pinned during the run system prompt SHA-256 of the exact text used tool definitions SHA-256 of the tool/function description bundle; enabled tool list guardrail config product + version + thresholds per category retrieval index identity + snapshot date + document count environment endpoint, tenant, auth mode, dates and times of each run ``` Digests rather than the text itself: they let anyone re-compute and compare without the appendix carrying the prompt into a document that circulates. The same appendix makes the engagement reproducible, which is the same property viewed from the other end. **Name the invalidating changes, mapped to conclusions** “Results may not apply if the system changes” is unactionable, because it delegates the judgement to the person least able to make it. List the events and say what each one kills: ```text VALIDITY Results describe Appendix A only. Voided by: model or serving-tier change (all sections) | system-prompt or policy edit (4, 6) | new tool or data source reaching context (5, 6) | guardrail config or threshold change (7) | new entry point (coverage declaration, 3). Re-test: on any trigger above, or 6 months, whichever comes first. ``` Mapping triggers to sections means a small change costs a partial re-test rather than discarding the report — which is what makes the rule survive contact with a team that ships weekly. **What it costs** The fingerprint is an hour at the end of the engagement, mostly digest computation and asking two questions of the platform team. A triggered partial re-test is the real expense: re-driving one section is typically half a day of setup plus the model calls for that section's attempts — hundreds of dollars, not tens of thousands — whereas a full re-run is the original engagement again. That asymmetry is the argument that sells the section mapping: without it, every prompt edit nominally invalidates everything, so in practice teams invalidate nothing. **Where the number misleads — two things a fingerprint cannot fix** First, an unchanged configuration does not give an unchanged result. Generation is non-deterministic and attack success is probabilistic, so a re-run scoring worse is not evidence of regression until you know the sampling variance. The arithmetic worth carrying: zero hits in 50 attempts is consistent with a true success rate up to roughly 6% (the rule of three, upper bound ≈ 3/n). A clean row was always an interval, never a zero, and per-item attempt counts are what let a reader see how wide that interval is. Second, hosted models change beneath a stable identifier. The provider may alter weights, serving stack or system-side policy without changing the string you recorded, so a matching model id proves less than it looks. Say what you could pin, and state provider-side drift as an explicit assumption whose only detection method is re-testing. **Make the process carry it, not the prose** Caveats do not survive being quoted; lookups do. Three mechanisms that work: a coverage ledger maintained across engagements rather than per report, so the organisation tracks which surfaces have ever been driven and when; a rule that any external assurance claim must cite a run identifier and its fingerprint, which forces someone to open the artefact; and a short front-of-report block written for whoever reads only page one. The report can make the lookup possible; only the process can make it happen. **What I check before sending** Can a reader re-compute every digest in Appendix A from artefacts they control? Does each trigger name the sections it voids rather than the whole document? Does every clean result carry its attempt count, so its interval is visible? Is provider drift stated as an assumption rather than assumed away? And is there any sentence in the report that would still read as an endorsement six months from now if quoted with the date stripped off?