A client tracks year-over-year AI red-team trends by published adversary-technique identifier, and the reference your team maps against was reorganised between last year's engagement and this one. How do you keep the two engagements comparable?

This is a measurement-integrity problem wearing a taxonomy costume. The client's real question is "are we getting better?" and technique identifiers are a shaky instrument for answering it, because the instrument's scale was redrawn between the two readings. ### What a revision actually does Reorganisations do three distinct things, and they are not equally dangerous. | Change | What it does | Risk | |---|---|---| | **Rename** | Same class, new code or new name | Low — mechanical, a crosswalk handles it | | **Merge** | Two classes become one | Low, but lossy in one direction: you can never recover which of the two an old finding was | | **Split** | One class becomes several successors | **High — this is the trap** | A split is the case a rename table cannot express. The table names one successor, and every historical finding under the old class is silently filed there, including the ones that belong under a sibling. Nothing errors; the counts just come out wrong, and they come out wrong in a way that looks like a real change in the client's posture. ### The three moves **Stamp the edition, at authoring time.** Every mapping records which reference and which edition produced it. This is cheap now and unrecoverable later: a year on, nobody can tell whether a code that no longer resolves was retired, renamed, or simply typed wrong. **Re-map from the original evidence, not from a crosswalk.** For split classes, only the finding's own evidence — attacker position, required access, resulting effect — decides which successor it belongs under. A supplied crosswalk, even an official one, cannot make that call for an individual finding, because it operates on classes and the ambiguity is per-finding. **Carry both identifiers** on every re-mapped item, so an auditor holding last year's PDF can reconcile it against this year's chart without asking you. ### What it costs This is the paragraph that decides the answer. Re-mapping scales with the **historical corpus**, not with this year's findings. A three-year history of two hundred findings, at ten to twenty minutes each to pull the evidence and re-derive the placement, is roughly five to ten analyst days — and it recurs on every revision, unbudgeted, with no new security value delivered to anyone. That is why teams defer it, and why the trend then quietly becomes uninterpretable. If you intend to trend on published identifiers, this cost is part of the programme, and it should be in the statement of work rather than discovered mid-cycle. ### Where the number misleads The specific failure is a dashboard that aggregates identifiers across editions with **no edition field**. Once that exists, a bar that moved because a class was split, merged or renamed is indistinguishable from a bar that moved because the system got better or worse — and the default reading in the room is always the second one. Two shapes to watch for: a category that drops to zero because its code was retired, read as "we fixed that"; and a category that doubles because a broad class was split and both successors now appear, read as a regression. Neither corresponds to anything that happened to the target. So the trend is drawn on the **re-mapped set only**, with an explicit note that the prior year was re-mapped and which classes moved, annotated on the chart rather than in an appendix. An unexplained jump gets interpreted as a security event; an explained one costs a sentence. ### The judgment call Technique catalogues were designed as a shared vocabulary, not as a metric, and using one as a metric is what creates this recurring bill. If the client's genuine question is programme effectiveness, a small set of **house categories you control** — stable by construction, revised only when you decide — plus published identifiers retained per finding for lookup and routing, is usually both cheaper and more honest. If their question is comparability with outside parties, a regulator, or another vendor, the published structure has to stay primary and you pay the re-mapping cost deliberately. What I would refuse: freezing the team on a superseded edition to protect the trend line. It buys one year of smooth charts and costs the mapping its entire purpose, because a reader who looks an identifier up now finds a description that no longer exists. ### What I would check Does any dashboard, spreadsheet or ticket field store a technique identifier without the edition it was assigned under? That single missing column is what converts a taxonomy revision into a fake trend, and it is far easier to add before the next engagement than to reconstruct afterwards.