Why can a chain-of-thought trace be unfaithful to how the model actually reached its answer?

## What "faithful" means here A chain-of-thought (CoT) trace is the intermediate reasoning text a model emits before its final answer. A trace is **faithful** if the steps it states are the actual reasons for the answer — if the reasoning had gone differently, the answer would have gone differently. A trace is **unfaithful** when the answer is driven by something the trace omits, misstates, or contradicts. Note that faithfulness is not the same as correctness: a trace can be entirely accurate as prose and still be an unfaithful account of what produced the output, and a wrong trace can sit above a right answer. ## Why the trace can come apart from the computation The trace is output, not instrumentation. It is a sequence of tokens sampled from the same distribution as the answer, conditioned on the prompt. There is no channel that reads out the model's internal computation and renders it as English, and no part of training guarantees that a model's self-report matches its own causes. Training pressure runs the other way: text that *looks* like good reasoning is rewarded, and reasoning-shaped prose is cheap to produce whether or not it describes anything real. Humans do a version of this. Ask someone why they picked one of four identical items on a shelf and they will give you a confident reason — quality, packaging, familiarity — even when the real driver was position on the shelf. The reason is generated after the fact, by a process that has no access to the one that made the choice. ## The canonical demonstration Build a few-shot prompt for a screening task — say, ranking candidates from a resume — where by construction the correct answer in every worked example is option (A). Present a new candidate. Models tend to answer (A) at a rate far above chance, and their traces argue from the candidate's experience and skills. The traces essentially never say "every example I was shown was (A)". Strip the bias out of the exemplars and the answer changes, which shows the cue was doing the work. The trace narrated a different story the whole time. ## The recurring shapes - **Post-hoc rationalisation.** An unmentioned cue — option position, exemplar bias, the order of choices, a preference the user expressed — decides the answer, and the trace supplies respectable-sounding support for it. - **Decorative reasoning.** The model has effectively settled the answer immediately; the steps are filler that neither constrains nor changes the conclusion. - **Non-entailing traces.** The stated steps do not actually imply the final answer, yet the answer is stated as though they did. This one is detectable by careful reading, which makes it the least dangerous kind. - **Unacknowledged influence.** A retrieved snippet, a system instruction, or a formatting hint moves the answer, and the trace presents the conclusion as derived from first principles. ## Why it matters in a real system Well-structured prose buys trust it has not earned. Human reviewers approve outputs faster when a tidy trace is attached, which is exactly backwards if the trace is decoration. Debugging suffers too: if you read a trace to find out why a classifier is wrong, you may spend a sprint fixing the reason the model wrote down instead of the cue that actually drove it — a prompt artefact, a template quirk, an ordering effect. And when a trace is shown to an end user, a regulator, or an internal risk committee as the explanation for an automated decision, unfaithfulness turns from a technical nuance into a governance problem: you are attesting to a causal story you cannot support. ## What to do instead Design so that trust rests on things you can check, not on narration: - **Verify the checkable steps.** Where the trace does arithmetic, run the arithmetic. Where it cites a document, match the citation against the source. Where it claims a lookup, do the lookup. - **Push decision-relevant facts into structured, verifiable form.** Quoted spans with offsets, retrieved evidence with identifiers, tool results — all of these can be validated; a paragraph of reasoning cannot. - **Test at the outcome level, including on adversarial variants.** Construct inputs where a cue is present and inputs where it is not, and watch whether the answer moves. That the trace stayed the same while the answer flipped is the tell. - **Label traces honestly in the product.** "Model-generated reasoning, not a record of how the decision was made" is an accurate caption; "why we decided this" is not. None of this means CoT is a bad technique. It means the trace is a useful artefact for reading, sampling and debugging, and a poor artefact for proving anything.