What makes speaker diarization fail on a three-way consultation with an interpreter?

## What diarization is, and what it is not Diarization answers **who spoke when**. It produces speaker-labelled time spans — `speaker_1: 00:12.4–00:19.8` — which are then joined to the transcript using word-level timestamps to yield a turn-attributed transcript. It is *not* speaker identification: out of the box the labels are anonymous cluster indices, stable only within a single recording. Mapping `speaker_1` to a named person requires enrolment (voice prints) or external context such as channel assignment or the meeting roster, and that is a separate, consent-sensitive step. ## The pipeline and its failure points **1. Segmentation.** Detect points where the active voice changes. Fails on overlap: when two people speak simultaneously there is no clean boundary, and most systems assign the span to a single dominant speaker or drop it. Overlap is where a large share of total diarization error lives, and it clusters exactly at the interesting moments — interruptions, corrections, disagreement. **2. Embedding.** Each segment is turned into a fixed-length speaker embedding. Fails on short segments: a 300 ms "mm-hm" produces a noisy vector. Back-channels are therefore routinely attributed to whoever was holding the floor. **3. Clustering.** Group embeddings into speakers, usually without being told how many there are. Over-clustering splits one person into several labels (common when someone changes register — reading a dosage aloud, raising their voice, or switching language). Under-clustering merges two similar voices, which is likeliest for same-gender, similar-age speakers on a noisy channel. **4. Alignment.** Join spans to words. Off-by-a-word errors at turn edges put the first word of a reply on the previous speaker. ## Why the interpreter case is the hard one A two-person recorded consultation is close to the easy case: two distinct voices, mostly alternating, few interruptions. Add an interpreter and three things change at once. The speaker count is now ambiguous to a system that has to infer it. The interpreter speaks in two languages, and speaker embeddings shift with language and prosody, encouraging over-clustering. And the turn structure becomes tightly interleaved with frequent overlap, because the interpreter starts before the speaker fully stops. If you also pass the audio through a language-detection step, a per-segment language guess can flip mid-turn and destabilize downstream processing. ## Word timestamps: the enrichment that makes the transcript reviewable Word-level timestamps are what let a reviewer click any sentence and replay exactly that audio. In a disputed clinical record that is the difference between "the transcript says X" and "here is the audio saying X". They are also what diarization joins against, what lets you highlight low-confidence spans in place, and what makes partial re-transcription possible — you can re-run a 4-second window rather than the whole file. Ask for them explicitly; some pipelines only emit segment-level times, which is too coarse for both uses. ## Measuring it honestly **Diarization error rate (DER)** sums missed speech, false-alarm speech and speaker-confusion time over total speech time. Two disciplines matter. First, many published DERs use a forgiveness collar around boundaries and *exclude overlapped speech*; if your product cares about interruptions, score with overlap included and say so. Second, report DER separately from WER — a transcript can be word-perfect and still attribute every sentence to the wrong person, and averaging the two hides which one is failing. Useful secondary metrics: **speaker count error** (did it find three speakers or five?), **turn-boundary F1**, and **word diarization error rate**, which scores attribution per word and is closer to what a reader experiences. ## Practical mitigations - **Separate channels beat any algorithm.** If each participant has their own microphone or telephony leg, diarization is a routing fact, not an inference. Take that whenever the deployment allows it. - **Give the number of speakers** when you know it; most APIs accept a hint or a min/max, and it removes the hardest guess. - **Fix the capture chain** — a far-field single mic in a reverberant room degrades embeddings more than model choice does. - **Smooth the output**: suppress single-word speaker flips, and snap boundaries to word edges. - **Design for uncertainty**: mark low-confidence attributions in the UI and let a reviewer reassign a turn, rather than presenting labels as fact.