You must keep a harmful model completion out of an AI red-team report, but the reader still has to believe the attack worked. What do you put in the report in place of the completion, and what does the reader lose by accepting it?

Redaction here is a **substitution** problem, not a deletion problem. Deleting the completion deletes the finding's evidentiary value along with it, so the work is to identify which properties of that text the reader actually needed and record those instead. ## What the completion was doing for the reader One question only: *did the model really do the harmful thing, or did it produce something that merely reads as though it did?* That distinction drives severity, and it is the thing a reader loses when the quotation goes. Fluent, confident, entirely useless text is common; so is text that is short, flat and genuinely actionable. Length, tone and vividness do not separate them. A characterisation must therefore state what made the output actionable — specificity, ordering, quantities, plausibility of correctness — without reproducing any of it. ## The four parts of a substitute record **Characterisation.** Prose written at the granularity of the claim. If the claim is "the model gave actionable uplift", the characterisation has to justify "actionable". If the claim is only "the refusal boundary failed", a much thinner description is honest and sufficient. Matching depth to claim is the skill; over-writing a weak finding is as much a defect as under-writing a strong one. **Provenance of the verdict.** Who or what decided this counted, and under what rubric. Either a human reviewer, identified by name, with the harm scale and level they applied; or an automated grader, named with its version and configuration, plus whether a human spot-checked its verdicts and at what sample rate. An unattributed verdict is the single most common weakness in redacted findings, because it collapses the whole finding into an anonymous assertion. **Reproduction parameters minus the payload.** Target build, system-prompt hash, decoding settings such as temperature and top-p, date, attack family, turn count, attempts and successes. This is what lets an engineer re-derive the result under supervision instead of replaying stored harmful text. **A seal.** A cryptographic hash of the raw artefact, its storage location, the access procedure and the retention date. The hash converts "trust my description" into "the artefact I described has not changed since I described it". ## What it costs, and what the numbers do not say Each substituted finding costs roughly half an hour of careful writing, and a two-reviewer rule on high-severity items costs a second reviewer the same again. Over a thirty-finding engagement that is a day of senior time spent producing no new findings — the price of the archive being safe to circulate. Then the misleading part, which matters more. A redacted finding is read almost entirely through two numbers, and both mislead by default. The **success ratio**. "13 of 20 attempts" is meaningless until the report defines an attempt. One prompt? One complete multi-turn conversation? One seed prompt sampled twenty times at nonzero temperature, where the variance is the sampler's and not the attack's? These give wildly different pictures of how reliable the attack is, and a reader who assumes the wrong one will mis-set severity in either direction. Write the denominator's definition into the record next to the ratio, always. The **grader's verdict**. If an automated judge decided the attempt succeeded, its own error rate is now part of your finding, and it is usually unmeasured. Judges over-count text that begins with a refusal and then complies, and under-count harm expressed obliquely. Reporting "the grader scored this a hit" without a human spot-check rate hands the reader a number whose accuracy nobody has established. Two more failure modes belong on this list. **Over-redaction flattens severity**: when every finding is described in the same guarded vocabulary, the client prioritises by your adjectives rather than by evidence, and genuinely severe items sink into the mass. **Fragments recompose**: a "short harmless excerpt" chosen to make one finding vivid is chosen precisely because it is the specific line, and excerpts across several findings can reconstitute more than any single one shows. Automated identifier scrubbers do not help here — they were built to remove names and account numbers, not to neutralise instructional content, so they return exactly the sentences a determined reader wants. ## What I would check Read the characterisation back looking for any string that could be lifted and used as-is. Confirm every finding's verdict names its source and rubric. Confirm the ratio is accompanied by a definition of an attempt. Have a second reviewer open the sealed artefact and confirm the description matches it — that check is the only thing standing between a redacted report and an unfalsifiable one.