Why is using the same model to propose and score Tree of Thought states risky?

## The setup that creates the problem The default Tree of Thought implementation uses one model twice: once to generate candidate next steps, once to score the states those steps produce. It is the cheapest arrangement and the natural place to start. It also makes the evaluator's judgement statistically dependent on the generator's, and search relies on that judgement being informative. The intuition is simple. If a model produced a step, it did so because that step looked correct to it. Asking the same model, moments later and often with the same reasoning in context, whether that step looks correct is not an independent test. Errors invisible during generation are usually invisible during evaluation, for the same reason: the model's understanding of the problem is what produced them. ## What it looks like inside the search Two symptoms show up, and both are observable without any special instrumentation. **Score compression at the top.** The distribution of values bunches into a narrow band near the high end. Very few states get decisively rejected. This is the more damaging symptom, because absolute thresholds depend on bad states actually scoring low — if nothing falls below the cutoff, pruning is decorative. **Weak separation between siblings.** All candidates at a node came from the same generator and share its assumptions, so to their author they look comparably reasonable. Rankings become close to arbitrary, which means the search's choice of what to expand carries little signal. The combined effect is the failure mode worth naming in an interview: the tree keeps its cost and loses its benefit. Branching is only worth its multiplied token spend because evaluation lets you concentrate effort on promising states and abandon the rest. An evaluator that approves nearly everything turns the search into breadth for its own sake — many times the cost of a single linear pass, with accuracy that barely moves. A related trap is that the search can lock onto one line of reasoning. If the generator commits to an approach early and the evaluator, sharing that commitment, scores every state along that line highly, alternatives that would have required abandoning the premise get scored down. The tree flattens into an expensive chain. ## How to detect it Do not take the scores at face value; test them against something you already know. Build a small set of states whose quality is known — some from problems that were solved correctly, some from branches that demonstrably failed — and run the evaluator over them. If the scores do not separate the two groups, the evaluator carries no usable signal for this task, whatever it says about individual nodes. Also look at the score histogram from real runs. A healthy evaluator produces spread, with a genuine low tail. A distribution squeezed into the top fifth of the scale is the compression symptom, and it usually means the threshold is either doing nothing or has been nudged so high that it is pruning on noise. ## What actually helps **Hide the trace.** Present the state to be judged on its own, without the generation reasoning that produced it. The rationale in context is a strong cue toward endorsement; removing it makes the judgement about the state rather than about the argument for it. **Change who judges.** A different model, or at minimum a different configuration, breaks some of the dependence. It is not a complete fix — models share training influences and often share blind spots — but it reliably widens the score distribution compared with pure self-scoring. **Prefer any external signal.** This is the strongest correction and the reason it dominates in practice. Where the task offers a check that is not the model's opinion — a test suite, a constraint solver, a compiler, a lookup against known reagents or known facts — that signal should dominate the score. Self-judgement fills the gaps it cannot cover, not the other way round. **Reframe as comparison.** Asking which of several states is most promising, rather than assigning each an absolute value, is somewhat more robust than absolute self-scoring, because relative judgements are easier. It does not remove shared blind spots, and it gives up the absolute floor that pruning wants. **Ask for the objection first.** Prompting the evaluator to state the strongest reason the state might fail, before it produces a verdict, raises the rate at which weak states are caught. It is a partial mitigation, not a substitute for an external check — the model still cannot object to errors it does not perceive. ## The honest summary Self-evaluation is not worthless; a model can often reject a state that violates a stated constraint or contradicts itself. It is simply weakest exactly where search needs it most — on subtle errors that the same model made a moment earlier. Design around that: use it as a cheap first filter, anchor it to external signals wherever the task provides them, and verify it separates known-good from known-bad states before trusting it to steer an expensive tree.