In a reconstruction built from repeated samples, does span agreement prove recall?

## The problem being solved You have been handed four hundred captured auto-replies from an unattended triage workflow and a reconstructed passage somebody assembled from them, said to be another customer's ticket - a name, an order number, dates, an internal resolution note. Your job is to say which parts are evidence and which parts are the model producing plausible text. Nothing in the capture distinguishes them: a recalled span and a confabulated span arrive in the same register, in the same reply shape, past the same screening layer. The only lever you have is repetition. Ask for the same region across many independent runs and look at where the samples disagree. ## Why the test works in one direction A span that is genuinely present in the run's context is being copied. Copying is a low-entropy operation for a model: across samples, the same characters keep coming back, because the context makes that continuation overwhelmingly likely. A span the model is filling in is being drawn from a much flatter distribution. There is no order number in front of it, so it produces something order-number-shaped, and next time a different one. Variation across independent samples is therefore strong evidence of generation. This is the reliable half of the test, and it is the half that lets you strike spans off a reconstruction with confidence. ## Why the reverse inference is weak Agreement establishes that the output distribution for that position is sharply peaked. It does not establish why. A distribution peaks for at least three reasons that have nothing to do with a source text being present. The span may be format-forced: a date in an obvious format, a reference number with a fixed prefix, a closing line the surrounding style makes near-inevitable. It may be a high-frequency default: a very common given name, a round quantity, a canonical phrasing. Or it may be shaped by the request itself, if the captured replies were elicited in a way that suggested the answer's form - in which case every sample agrees because every sample was pushed the same way, and the agreement measures the request, not the context. So the correct reading is: disagreement rules a span out, agreement fails to rule it out. That is much less than confirmation, and a reconstruction presented as verified because it was stable is overclaiming. ## The two conditions that make the test usable at all First, the decoder has to vary. If the deployment decodes near-deterministically, every sample agrees on everything and the signal is identically zero - you will observe perfect stability across a recalled span and an invented one alike. A triager who reports high sample agreement without knowing whether the deployment samples at all has measured nothing. Where the decoder cannot vary, any variation has to come from the request side, and variation introduced that way changes the conditions between samples, which weakens the comparison. Second, independent has to mean separate runs. Asking twice inside one context is not two samples: the second answer is conditioned on the first, which is the strongest possible pressure toward agreement, and an invented span becomes self-consistent the moment it is on the page. ## What actually confirms a span Only something outside the captured set. Checking a recovered order number against the real record confirms it. Structural corroboration helps a little - a recovered date that is consistent with a recovered reference number, both stable, is better than either alone - but consistency is also what a fluent generator produces, so this raises confidence rather than settling it. In practice a triaged reconstruction ends up in three buckets: confirmed against a source, struck out by disagreement, and stable-but-unconfirmed. The third bucket is usually the largest and it is the one people quietly promote into the first. ## The trap worth naming in an interview The spans that most reward stability testing are the ones the test handles worst. Names, dates and identifiers are the payoff of the whole exercise and also the most format-constrained things in the passage - the categories where a model converges on a plausible value with no source at all. The prose in between, which nobody cares about, is where sample agreement is actually informative. ## What an interviewer is listening for The asymmetry, stated as an asymmetry; the decoder caveat; the difference between separate runs and repeated turns in one context; and a willingness to leave spans in the unconfirmed bucket rather than rounding stability up to proof.