A directive found in one call transcript won't reappear on re-transcription — is that a finding?

## Why this comes up at all In a call-recording assistant, non-determinism sits in an unusual place. People expect the *model* to vary between runs. Here the variance is upstream: the recognition stage guesses over an ambiguous signal, so the same stored bytes transcribed twice can yield different text. The consequence is a triage situation that catches people out — a report arrives saying a spoken directive steered a drafted recap, the engineer re-runs the pipeline over the same recording, and nothing happens. ## The two questions inside one **Did it happen?** This is decidable and it is about records, not replays. The transcript the drafting stage actually consumed, the draft that came out, and its recipients are the evidence. In this setting the interesting failure is usually the audience: content from an adjacent recording or from an earlier call in the same account thread showing up in a recap that went to the wrong side of the table. That either did or did not go out. A fresh recognition run cannot un-send it. **Will it recur?** This is a rate, and one observation is a terrible estimator of a rate. Worse, the variance has two independent sources stacked — recognition and generation — so a single failed replay is weak evidence in both directions. If you need a rate, you need many trials and you should say what your denominator is, rather than reporting `reproduces intermittently` and leaving the reader to guess. Conflating the two is the actual error. Reports get closed as `not reproducible` when the question that mattered was already answered by the output records. ## What each artefact proves, exactly - **The stored audio** proves what was uploaded. It does not prove what text existed downstream, because at that point no text existed. - **A re-run transcript** proves what recognition emitted *this* time. It is a new object, not a measurement of the old one, and its disagreement with the first transcript refutes nothing. - **The used transcript** proves what the drafting stage was handed. It does not prove that any person uttered those words — recognition can emit text with no counterpart in the input. - **The draft and its recipients** prove what was produced and where it went. This is usually the strongest artefact you have, and the one that decides the exposure question. - **A pipeline run identifier** proves which components and versions were in the path, which matters because a component swap can change the behaviour without anyone touching the application. Notice how none of these prove intent, and be careful about asserting it. The direction of every claim has to be right: a directive in a transcript proves the string reached the drafting stage, not that somebody planned to put it there. ## Writing the finding State the class, not the string. The class is: the pipeline's inspection point sits before the artefact that carries text, so anything that reaches the model through recognition was never subject to it. Report the observed instance as one instance, name explicitly what you could and could not replay, and give the exposure separately from the reproduction rate. A report that leads with `reproduced 1 of 5 times` invites the wrong conclusion; a report that leads with `a recap containing another participant's material was drafted and sent, and here is the transcript it was drafted from` does not. If the transcript in question was not retained, say that plainly and reason from the draft. It is a weaker finding and it is still a finding. What is not acceptable is presenting a replayed run as if it were the original, or presenting a failed replay as a refutation. ## The judgment an interviewer is listening for They want to hear that you know a probabilistic pipeline does not yield binary reproduction, that you can name which artefact answers which question, and that you will not let `it did not reproduce` close a case where the output records already settled the exposure. They also want to hear the reverse discipline: one success against one deployment on one day is not a reliable finding either, and you should not inflate it into one.