How do you apply self-consistency when the sampled answers are free-form text?

## Why the standard recipe stops working Aggregation by counting rests on an equality test. For a number or an option label, canonicalisation makes that test meaningful. For eight free-text incident postmortem summaries, it does not: no two will be byte-identical, no normalizer will make them so, and every bucket has size one. Agreement between prose answers is semantic, and semantics needs a judge. ## Universal self-consistency The direct adaptation is to keep the *idea* of "pick the answer the others support" and change the *mechanism* from counting to model judgement. All candidate responses are placed in one prompt, labelled, and the model is asked which single response is most consistent with the rest. Its output is an index, not new text. The properties worth stating in an interview: - **No extractor.** Nothing has to be parsed out of the response, which is what makes it applicable to summaries, explanations, code and plans alike. - **Selection, not synthesis.** The winner is one of the sampled responses verbatim, so it is internally coherent and reads naturally. Nothing is stitched together. - **A second call.** You pay one extra inference over a prompt that holds every candidate, and long candidates multiply quickly against the context limit. On eight substantial postmortems this is the binding constraint, and it forces either truncation, a two-stage tournament, or fewer candidates. ## Alternatives worth knowing **Cluster-and-pick.** Represent each candidate as an embedding or compare them pairwise with an entailment check, cluster, and return the member of the largest cluster closest to its centre. This recovers something much closer to the original vote — the largest cluster *is* the bucket — and it gives you a numeric agreement measure, which the model-judge version does not. It is weaker when the candidates differ in ways the representation does not capture, for example two summaries that agree on wording but disagree on a root cause. **Claim-level voting.** Decompose each response into atomic factual claims, count claims across responses, keep the claims that most responses assert, and regenerate a response from the surviving claims. This is the only variant that can beat every individual candidate, because it can combine a fact from one with a fact from another. It is also the riskiest: decomposition is lossy, the re-synthesis step can assemble a combination no candidate endorsed, and consistency between claims is not checked by the vote. **Hybrid.** For structured free-form output — a postmortem with a fixed set of sections — you can vote exactly on the extractable fields (the identified root cause, the affected component, a numeric duration) and use universal self-consistency only on the narrative. This gives you a hard agreement number where one exists and a judged selection where it does not. ## Failure modes to name - **Shared blind spot.** If none of the eight postmortems identifies the real trigger, no selection method will produce it. Selection is bounded by the sample set. - **Judge bias.** A model choosing among candidates shows position effects and a preference for longer, more assertive text. Shuffling candidate order across repeated selections and checking whether the choice is stable is a cheap sanity check. - **Consensus toward blandness.** "Most consistent with the others" favours the response that commits to the least. A candidate that alone identifies a subtle cause is by construction the least consistent one, so this method can systematically discard the best answer on exactly the cases you cared about. - **No agreement number.** Unlike counting, the model-judge version returns a choice with no natural measure of how contested it was, which removes the abstain signal that makes the exact-match version safe to deploy. Recovering it — by clustering as well, or by asking the selector to rate agreement — is worth the extra work. ## The judgement to show The honest position is that free-form aggregation is weaker than exact-match aggregation, not merely different. Where you can force the task to yield an extractable answer — a score, a category, a decision — do that and vote on it, and treat prose selection as the fallback for the part of the output that genuinely resists it.