You lead an AI red team whose output must land inside a recognised AI risk-management framework's structure, so findings become entries against governance functions and control statements. How far do you restructure the findings to fit that structure, and what do you refuse to convert?
answer
- two layers, one source of truth
- convert the summary, never the evidence
- unmapped item beats a forced fit
- populated rows imply assessed rows
- budget the conversion, or caveats get cut
basics
~20 sRestructure the summary layer fully — risks, controls, owners and dates in the framework's language, because that is what gets funded. Refuse to convert the evidence: attack narratives, per-family rates and the scope of what was untested stay in a linked technical record. Never invent a control row to make an unmatched finding disappear.
solid answer
~50 sTwo layers, one source of truth. The **framework layer** is fully restructured: findings are grouped into risk statements, attached to control statements, given owners and dates, and written in the framework's vocabulary. Resisting that is how red teams get ignored — governance functions cannot act on an artefact shaped like a pentest report. The **evidence layer** is not restructured at all. Attack narratives, the mix of attack families and their separate rates, the target configuration, and an explicit statement of untested surface stay as they are, addressable by identifier. What I refuse: forcing a finding into a control it does not really bear on, dropping findings with no matching control, and letting a green framework view stand without a coverage caveat. The last is the real risk — a fully populated framework structure reads as completeness, and completeness is precisely what a time-boxed engagement cannot claim.
go deeper
Understands that the same findings get rewritten for a governance audience and that the technical detail should still exist somewhere.
Separates the summary layer from the evidence layer and keeps identifiers linking them, so the two do not drift apart.
Names what must not be converted — narratives, per-family results, untested surface — and refuses to force a finding into an ill-fitting control.
Sets the policy and its economics: who owns thresholds and acceptance, how the framework layer is generated rather than retyped, that unmapped findings are reported as framework gaps, and that a populated view without a coverage statement is a misrepresentation.
### The tradeoff being tested Restructure too little and the work never reaches the people who allocate budget: a governance function cannot act on an artefact shaped like a pentest report, so it gets filed and the findings die. Restructure too much and the framework view becomes a grid of green cells that overstates what was tested. Both failures are common. A principal-level answer names both rather than defending the technical report. ### Where the line goes **Fully convert — the framework layer.** Risk statements, control attribution, severity, owner, date, remediation status. This layer should read in the framework's own vocabulary, not ours. It is the layer executives read and fund. **Never convert — the evidence layer.** Attack narratives, per-attack-family results, the target and guard configuration under test, and an explicit statement of untested surface. Compressing these is what makes a later challenge unanswerable, because the only record of what was actually done has been paraphrased away. **Convert with an explicit note — measurements that become verdicts.** A pass may live in the framework layer provided the threshold, its owner, and the measurement's basis are recorded in the evidence layer and addressable by identifier. ### The three states a control can be in The mechanical detail that decides whether the view is honest: a control statement has **three** states, not two — satisfied, not satisfied, and *not assessed*. A view that offers only pass and fail forces every untested control into a blank cell, and a blank cell in a grid of greens reads as green. Making "not assessed" a first-class, visibly distinct state is the single change that does most for the artefact's honesty, and it is usually resisted because it makes the view look incomplete. The view *is* incomplete; that is the finding. ### What I refuse to do 1. *Force an unmatched finding into the nearest control.* It misdescribes what was seen and routes the item to an owner who is not responsible for it, so it will not be fixed. Carry it as an explicitly unmapped item with a proposed control gap — a framework that cannot express a real finding has told you something about the framework. 2. *Drop findings with no home.* The same failure, quieter, and harder to detect later. 3. *Publish a framework view without a coverage statement attached to the view itself.* One sentence naming the interfaces, model configuration, attack families and dates exercised, and what was not, prevents the whole artefact from being read as assurance it cannot support. ### What the conversion costs Budget it explicitly, because unfunded conversion is done the night before the committee meeting, and the caveats are the first thing cut. For a report with roughly twenty findings, expect a couple of days of senior time: writing risk statements in the framework's language, agreeing which control each bears on, finding accountable owners, and getting thresholds signed off. Across a portfolio of AI systems on a quarterly cadence that is a standing headcount cost, not a rounding error, and it competes directly with testing time. A team that has not costed it will quietly stop doing the evidence layer properly, because the framework layer is the one with the deadline. ### Where the framework's own numbers mislead The completeness metric is the trap. "Eighty percent of controls satisfied" has the framework's control catalogue as its denominator, not your system's attack surface. Those are unrelated populations: a framework may have many controls bearing on a component you never touched, and few bearing on the one interface you spent the engagement inside. A high percentage can therefore be produced by an engagement that exercised almost nothing, and the number moves when the framework is revised even though no testing happened. The second trap is the trend line. "Risks closed this quarter" counts register rows, and register rows are a triage artefact; re-grouping findings changes the count without changing the system. Any quarter-on-quarter comparison needs the grouping rule, the suite and the target configuration to have held constant, and someone has to say so in writing. ### How I would know it went wrong A committee member describes the system as "assessed" when one interface was exercised. A control shows satisfied while the evidence behind it has quietly expired against a floating model identifier. A re-test cannot be compared with the previous one because nobody kept the suite and configuration. Each traces back to a conversion decision made without a written rule — which is why the framework layer is generated from the evidence records rather than maintained by hand, and why every finding carries a stable identifier that appears in both.
- A finding matches no control statement in the framework. What do you do with it?Carry it as an explicitly unmapped item with a proposed control gap. A framework that cannot express a real finding has revealed a gap in itself, and hiding the finding in the nearest adjacent control loses both the issue and its owner.
- How do you stop the framework view from being read as full assurance?A scope statement attached to the view itself, not buried in an annex: which interfaces, which model configuration, which attack families, over what dates, and what was not exercised. Populated rows otherwise read as assessed rows.
- Who should write the framework layer — the red team or the governance function?The red team supplies structured findings and their limits; the governance function owns the framework vocabulary, thresholds and acceptance. Generating the rows mechanically from the finding records keeps both honest and prevents drift.
saying these in an interview costs you the question
- Refusing to produce a framework view at all because it 'loses the technical truth'.
- Forcing every finding into some control so no row is empty.
- Presenting a fully populated framework view as evidence the system is assured.
- Maintaining the framework layer by hand, separately from the evidence.
- Letting the red team set acceptance thresholds with no accountable risk owner.