Why does self-consistency break down if you sample the chains at temperature 0?

## What self-consistency asks of the sampler Self-consistency replaces a single greedy chain-of-thought with n independently sampled chains and keeps the final answer most of them agree on. The statistical bet is that a correct line of reasoning can be reached by several different routes, while wrong routes scatter — errors are only partially correlated, so the modal answer is more reliable than any one chain. Everything in that bet rests on the samples being **quasi-independent**. If the n chains are the same computation repeated, the vote carries exactly one bit of information and the extra spend buys nothing. ## Why temperature 0 degenerates Temperature rescales the logits before the softmax. As it goes to 0 the distribution collapses onto the argmax, which is greedy decoding: at every step the same token wins, so the whole trajectory is fixed by the prompt. Sample forty chains on the same orbital-mechanics word problem at temperature 0 and you get forty near-identical traces — same setup, same intermediate quantities, same arithmetic slip if there is one. The answer histogram is a single spike, the vote is trivially that spike, and the accuracy equals plain greedy chain-of-thought at forty times the token bill. A useful smoke test before trusting any self-consistency deployment: count distinct final answers at n=20. If it is almost always 1, the sampler is not doing its job. ## The knobs that actually create paths - **Temperature** decides how much probability mass reaches non-argmax tokens. It is the primary diversity control, because it operates at the decision points where the model chooses a strategy, sets up an equation, or picks a unit conversion. - **Nucleus (top-p) truncation** keeps only the smallest set of tokens whose cumulative probability reaches p and renormalizes over them. It bounds how bad a sampled token can be, which matters because raising temperature alone eventually admits incoherent continuations. Related truncation schemes such as typical sampling and min-p trade off the same way. The two interact: temperature says how flat the distribution gets, truncation says how far into the tail you are allowed to reach. Reported results for self-consistency are fairly robust across a band of temperatures rather than requiring a precisely tuned value, but the band is model- and task-specific — measure it. ## Surface diversity versus path diversity This is the distinction candidates most often miss. Two chains can differ in every sentence and still perform identical reasoning: same decomposition, same equation, same numbers. That is paraphrase, and it votes as a single chain. On an internal arithmetic word-problem set, comparing top-p 0.95 against typical sampling looked equally diverse when scored by wording variance, and clearly different when scored by distinct intermediate quantities and solution strategies. Wording varies on filler tokens, which are the cheapest ones to randomize; the tokens you want varied are the ones that commit the chain to a route. ## How to measure it - Number of **distinct normalized final answers**, or the entropy of the answer distribution, across n samples. - Number of **distinct intermediate-step signatures** — the sequence of equations, chosen operations, or retrieved facts, normalized before comparison. - Surface metrics such as edit distance or self-BLEU between rationales, treated only as a weak proxy; they cannot distinguish paraphrase from a different route. - The decisive check is downstream: vote accuracy on a labelled slice versus greedy accuracy. Diversity that does not move that number is not worth paying for. ## The quality–diversity tradeoff Raising temperature raises the chance that some chain finds the right route, and lowers the chance that any given chain is coherent. Vote accuracy is a product of the two, so it traces an inverted U: it improves over greedy up to some temperature, then falls as chains become sloppy. Sweep temperature at fixed n on a held-out labelled slice and take the peak, rather than importing a number from a paper written about a different model. ## Levers beyond the decoding knobs Prompt-level variation often produces more genuine path diversity than temperature does: shuffling few-shot exemplars, instructing different solution strategies per sample, or varying the output format pushes the chains apart at the decision points rather than at the adjectives. This also matters for reasoning-mode endpoints, where providers may restrict or ignore sampling parameters and instead expose a thinking budget — there you get diversity from prompt variation and from the model's own stochastic reasoning, not from a temperature dial you control. Check what the mode you are calling actually honours before assuming the knob exists. ## Common failure modes Sampling at temperature 0 or near it "for stability"; declaring diversity from eyeballing prose; cranking temperature past the coherence cliff and reading the resulting accuracy drop as evidence that self-consistency does not work; and never checking the distinct-answer count, which would have revealed the problem in one line of code.