What must you record about an AI red-team run so that repeating it months later is a comparison against the filed result rather than a fresh experiment?

The manifest exists to make a future disagreement decidable. Six months from now two people will point at two numbers and argue about what changed; everything not written down at run time becomes opinion. A success rate with no configuration attached is not a result, it is an anecdote with a decimal point. ### What goes in it A workable shape, written by the harness itself and stored beside the raw transcripts: ```yaml run_id: 2026-04-11-full-suite target: endpoint: chat.internal/v1/completions model_requested: <pinned dated identifier> model_metadata_returned: <as returned, verbatim> gateway: <proxy name/version, if any> sampling: temperature: 0.0 top_p: 1.0 max_tokens: 1024 seed: 12345 # if honoured; record that it was not, if not application: system_prompt_sha256: <hash> tool_defs_sha256: <hash> retrieval_snapshot: <corpus version or date> guards: input_classifier: <name/version, thresholds, fail-open?> output_classifier: <name/version, thresholds> instrument: harness_version: <version> attack_prompt_set_sha256: <hash> generated_by: <generator + seed, if dynamic> decider: <the component judging a hit: version; if a model, its id + prompt hash> execution: trials_per_prompt: 20 attempted / completed: 4400 / 4400 metered_cost: <units> ``` Five groups, and each earns its place. **Target identity** says what answered, including the gateway, because a proxy in the path can rewrite metadata and route requests. **Sampling settings** matter because comparing two runs at different temperatures measures the temperature. **The application around the model** — prompt hash, tool definitions, retrieval corpus state, guard versions and thresholds — is where most silent change actually happens. **The instrument** is the half people forget: the harness, the exact attack prompt set, and the component that decides whether an attempt counted. **The evidence** is trials per prompt plus the raw transcripts, because aggregates cannot be re-examined. ### What it buys *Comparability*: a rerun whose manifest differs is a different experiment, and the manifest tells you which line moved. *Re-scoring*: with transcripts retained you can re-decide the OLD run with the CURRENT judge, which cleanly separates judge drift from target drift — the single most common false regression in this work. *Budgeting*: recorded request counts and metered cost are what let you tell a stakeholder what an out-of-cadence rerun will cost before you call one. ### What it costs Recording is cheap in money — a full suite's transcripts are tens of megabytes — and expensive in policy. Raw transcripts of a red-team run contain the generated harmful content the run elicited, so they need encryption, access control, a named retention window and a defensible answer about who may read them. The cost of *not* recording is larger and lands later: an unreproducible number means a full metered rerun to settle an argument, at exactly the moment someone is waiting on a release decision. ### Where the number misleads A rate quoted without its denominator is the commonest trap: "12% success" is 3 of 25 or 120 of 1000, and those two carry wildly different intervals while printing the same digits. Next is the partially completed run — rate limits, a timeout or a truncated sweep — reported against the full-suite denominator, which understates the rate and overstates the coverage simultaneously. Then judge drift: if the decider is itself a model or a versioned classifier, the rate can move with the target untouched, and without transcripts you cannot tell. Then the regenerated prompt set: a harness that samples fresh attack prompts each run changes the measuring instrument between runs, so no two rates share a denominator. Finally the false determinism claim — a `seed` parameter recorded in the manifest but never honoured by the endpoint is worse than no seed at all, because it converts an unknown into a wrong belief. ### What you check Recompute the hashes at run time rather than copying them forward from the previous manifest — a stale copied hash is how a changed prompt gets recorded as unchanged. Confirm the response metadata was captured verbatim rather than the value you requested. Verify the seed was honoured by issuing one duplicate request. Check that attempted and completed counts match, and that the denominator in any reported rate is the completed count. And generate the manifest from the harness, never by hand: a manifest a human types is a manifest that describes what someone believed they ran.