Why do off-the-shelf PII detectors miss hospital MRNs, and what does over-redaction cost?

## What a detector actually is PII detection combines several weak signals. Deterministic recognizers match a pattern and optionally validate it — a card number matches a digit shape and then passes a Luhn check; a national health number matches ten digits and then passes a modulus-11 check digit. Context words near the span raise the score. A named-entity model handles the things with no format at all: person names, organisations, locations. An anonymizer then applies an operator per entity type — replace, mask, hash, encrypt. Every part of that is probabilistic. The output has a recall figure (what fraction of real identifiers were caught) and a precision figure (what fraction of flagged spans were really identifiers), and neither is 100%. That framing matters more than any implementation detail, because it tells you redaction cannot be the thing that carries an access-control decision. ## Why domain identifiers are the ones that leak Shipped recognizers exist where a format is standardised across a country or an industry, because only then can one regex generalise. A medical record number has no such standard: each facility chooses its width, prefix and zero-padding, and the same digit string means something different at the hospital next door. Internal case numbers, claim numbers, accession numbers, device serials, staff badge numbers and account references behave the same way. These are the highest-value misses. Precisely because they are internal, anyone inside the organisation with access to the record system can resolve one back to a person — so a "de-identified" note that still carries an MRN is not de-identified at all for the population most likely to read the logs. Free text is where they hide. Structured fields can be handled by schema; a note reading "discussed with the family, chart 0042731, Tuesday" needs the detector to fire mid-sentence. Real clinical text also brings hyphenation, OCR noise from scanned documents, identifiers broken across line ends, and abbreviations that appear nowhere in the training data of a general-purpose name model. Person-name recall in that setting is materially worse than the benchmark numbers you will see quoted, which were measured on clean edited prose. ## Building recall you can defend Register custom recognizers for each identifier family your domain uses: a regex, a validation function where a check digit or range exists, and context words. Give each a score you can tune, and prefer a lower threshold with a validator over a high-confidence bare regex. Then measure, and measure the right thing. Label a sample of real production documents span by span, and report recall and precision **separately for each entity type**. Aggregate accuracy is close to useless here, because email addresses are easy and dominate the count while the MRN you actually worry about is rare. Track a leak-through rate — identifiers surviving per thousand documents — because that is the number a risk owner can reason about. Keep an adversarial slice: OCR artefacts, digits written as words, identifiers split by newlines, unusual and transliterated names. Re-run the suite whenever recognizers or the underlying model change, since a detection pipeline silently regresses like any other classifier. ## The other direction, and why it is not free Recall is easy to buy if you stop caring about precision: mask every digit run and every capitalised token. What that destroys is the content the task depended on. In a clinical note the digits are dosages, lab values, gestational ages, dates and intervals; strip them and the assistant produces confident, fluent summaries that are wrong. Mask all names and a note discussing two patients collapses into ambiguity that the model resolves by guessing. There is a second-order cost that decides most real programmes. If the redacted tool gives poor answers, clinicians stop using it and paste the same note into whatever consumer chatbot is on their phone — an unmonitored, unlogged, entirely uncontrolled path. Over-redaction can therefore produce a worse privacy outcome than the leak it prevented. Utility is a safety property here, not a competing concern. ## Reversible placeholders The usual escape is to replace rather than destroy: substitute stable placeholders (PATIENT_1, MRN_1, DATE_1) held in a server-side map for the life of the request, and rehydrate the model's output afterwards. This preserves referential structure — the model can still tell that the same person appears three times — and keeps relative dates usable, while the sensitive values never enter the prompt, the logs or the cache. The cost is holding the map, which is itself sensitive and short-lived by design, plus the rehydration step becoming a correctness dependency. ## Where redaction belongs in the stack Redaction reduces the blast radius of copies you cannot eliminate. It does not decide who may see what — isolation and access control do that below the model — and it does not stop a determined extraction. Order the layers: minimise what is fetched at all, redact what remains, isolate the corpus by tenant and role, and keep prompt bodies out of default logging. Redaction sitting alone as the only control is the configuration that ends up in an incident report.