Does a confident chain-of-thought trace mean its intermediate steps are true?

## The claim being tested Step-by-step output feels like a proof. It is not. A chain of thought is a sequence of generated sentences, produced by the same next-token process that produces the final answer, under the same pressure to be fluent and plausible. Nothing in that process checks a step against the world before writing it down. So an invented step is not an exotic failure — it is the ordinary failure mode of generation, wearing the costume of a derivation. ## Two concrete shapes **Invented computation.** Ask a model whether 17077 is prime. A plausible-looking answer is "17077 = 113 x 151, so it is composite." Multiply it out: 113 x 151 = 17063, not 17077. And 17077 has no divisor up to its square root, so it is prime. The chain is confident, it is formatted as arithmetic, and it is simply wrong at the step that carries all the weight. **Invented sources.** Ask a yes/no question about a supplier contract — may the buyer terminate for convenience? — and a trace may reason "under section 4.3(b), the buyer may terminate on 30 days' notice, therefore yes," where the document contains no 4.3(b) at all. The citation looks like grounding. It is decoration. Other members of the family: a lookup that returns a fact the source never stated, a unit conversion done with a made-up factor, a definition attributed to a standard that defines nothing of the sort, an intermediate quantity carried forward that was never derived. ## Why fabricated steps are more dangerous than bare wrong answers Three reasons compound. First, **structure buys trust**. A numbered chain reads as work shown. Reviewers who would push back on a naked assertion nod along to a derivation, because the derivation implies a process was followed. Second, **errors propagate**. Once a bad intermediate value or a bogus clause is written down, the rest of the chain conditions on it. The model does not go back; it builds. So a single fabricated step usually determines the final answer rather than getting averaged away. Third, **the answer can be right anyway**, which is worse for your evaluation loop than being wrong. If a model claims a nonexistent clause and still concludes "yes, termination is allowed" because some other clause happens to permit it, outcome-level testing scores a pass and the defect stays invisible until a case where the same habit lands on the wrong side. ## How to catch them The unhelpful move is to ask the model to double-check itself, or to ask for confidence per step. Self-reported confidence is generated text too, and it correlates with fluency more than with truth. The useful moves are external and mechanical: - **Recompute anything arithmetic.** If a step multiplies, divides, sums or converts, run it. Delegating the computation to a calculator or code path removes the failure entirely rather than detecting it. - **String-match every citation against the source.** A trace that cites section 4.3(b) should be rejected automatically when "4.3(b)" does not appear in the document. This is a cheap, deterministic check that catches an entire class of fabrication. - **Require quoted spans, not paraphrase.** Ask for the verbatim text the conclusion rests on, then verify the quote appears in the source. A model that must quote has far less room to invent. - **Verify the direction, not just the value.** If the chain says "therefore composite", the check is not only "are these factors real" but "does their product equal the input". ## The right mental model Read a chain of thought as a draft argument written by someone knowledgeable, fast, and occasionally willing to make something up rather than say "I don't know". You would not sign off on that person's memo without spot-checking the numbers and pulling up the cited section. Apply exactly that habit, and prefer to design the system so the checkable parts never depend on the model's word in the first place.