How do you run a closed-verdict audit without it becoming analyst performance management?

## The tension A verdict audit re-opens work that a named person did and, sometimes, concludes they were wrong. That is structurally an examination of an individual, however you frame it. But the moment it *is* one, it stops producing the data it exists to produce. This is the central design problem of the practice, and it is a leadership problem rather than a technical one. ## Fix the unit of analysis first Every finding should be expressed as a property of the **content or the process**, not of a person: - this rule cannot be dispositioned correctly without an authorised-activity reference nobody maintains; - this playbook licenses a close on a signal that does not support one; - this enrichment step, absent at the moment of decision, would have changed the verdict; - this log source was unavailable for the hour in question; - this disposition option does not exist in the case system, so analysts pick the nearest wrong one. All five are real findings, all five came from someone's case, and none of them names anybody. The reports are aggregate; the artefacts that go to the wider organisation contain rules, counts and fixes. ## Blind it in both directions Reviewers should not see who closed a case - partly for objectivity, partly so the exercise cannot be perceived as targeted. And the sample itself must be drawn by a rule, not by hand: a sampling method published in advance is your defence against the accusation that you went looking for one person. ## Agree the boundaries before the first cycle Write down, and get managers and HR to sign off, what the results feed and what they never feed. In practice: aggregate findings feed detection engineering and training content; they do not feed a performance rating, a bonus calculation or a disciplinary file. Some jurisdictions and worker-representation arrangements make monitoring of individual work product a consultation matter in its own right, so the legal and HR check belongs at design time, not after. The timing matters more than the wording. Negotiate this while the programme is hypothetical. After the first cycle turns up a run of dismissed cases that should have been escalated, every clause you try to add will look like a cover-up to one side and an obstruction to the other. ## The behaviour you are trying not to buy If analysts conclude their closures are being scored individually, three things follow, all measurable and all bad: 1. **Escalation inflation.** The safest personal move becomes escalating everything. Tier two drowns, the queue depth rises, and real signals are buried in defensive escalations. 2. **Notes written for the auditor.** Case notes become justifications rather than handover material, which degrades exactly the artefact the next responder depends on. 3. **Quiet non-cooperation.** People stop volunteering the cases they were unsure about - which are the highest-value cases the audit could possibly see. A verdict audit is a voluntary-cooperation instrument dressed as a measurement. Spend it once as a disciplinary tool and you do not get it back. ## But do not pretend the individual never exists The over-correction is equally dishonest. If repeated cycles, confirmed by a second blind reviewer on fresh samples, keep tracing overturned verdicts to one person, that is real and it matters - the queue is where intrusions are missed. The right handling is procedural: it goes to that person's manager through the normal coaching route, informed by the audit but not conducted by it, and only after you have ruled out the alternatives that would hit anyone in that seat - the shift they work, the rules they are assigned, a training gap, a playbook that told them to do it. The audit programme supplies evidence; it is not the disciplinary instrument, and it should not be the body that decides. ## Make cooperation pay The most effective protection is that the programme visibly repairs the job. Every finding gets a named owner and a date; you publish what changed - the reference data now in place, the playbook rewritten, the auto-close added, the new disposition option. Analysts cooperate with an audit that removes the cases they hated working. They defend themselves against one that only produces a percentage. ## The mirrored case: auditing a provider When a co-managing provider closes tickets on your estate, the same problem arrives without the culture lever - you have no relationship with their analysts and no standing to coach anyone. The contract has to do the work that norms do internally: a **right-to-audit** clause specifying sample size and cadence, access to the *raw evidence* behind each closure and not merely the ticket text (you cannot re-review what you cannot see), a named adjudicator for disputed verdicts, and what remediation is owed when a sample fails. The failure mode is the same shape: a provider who believes sampling threatens the contract will optimise the tickets rather than the verdicts. Negotiate it at signature or renewal; until then, the only real alternative is running your own detections over the same telemetry and comparing conclusions.