How do you choose the pruning threshold for Tree of Thought state scores?

## The decision the threshold makes Pruning throws away a partial state permanently. Every discarded state is compute you do not spend and, occasionally, a correct answer you will never reach. The threshold is where you place that trade, and the only honest way to place it is to measure both sides. Note the boundary: how many survivors the search keeps at each level, and in what order it visits them, is search configuration. The threshold question is narrower — where the cutoff sits on the score scale, and what it costs in recall. ## Building the curve The procedure is straightforward and almost always skipped. Take a set of problems whose correct answers you already have. Run the search with pruning disabled or set very permissively, so nothing is discarded prematurely. Log every state, the score its evaluator gave it, and — once the run finishes — whether the branch containing that state reached a correct final answer. Now sweep candidate cutoffs. For each one, compute two quantities: the fraction of nodes that would never have been expanded, and the fraction of problems that would no longer be solved because the winning branch passed through a state scoring below the cutoff. Plot them against each other and the shape of the trade becomes visible. The curve is usually not linear. There is often a flat region where raising the cutoff removes large numbers of nodes at almost no recall cost — those are the states the evaluator confidently and correctly rejects — followed by a knee where each additional saving starts costing real solutions. You want to sit just below the knee. Concretely: pruning every branch scoring below 0.3 might cut expansions by half while losing two percent of solved problems. That is a trade most teams take. If the same cutoff loses fifteen percent, it is not, and you either move the cutoff or improve the evaluator. ## Why the number does not transfer Scores produced by a prompted evaluator are not calibrated probabilities. A state scored 0.7 is not seven-times-in-ten likely to succeed; the number is an artefact of the scale you asked for and the model's habits on that scale. Two consequences follow. First, the cutoff is specific to the evaluator, the prompt, the model version and the task. Change any of them and the measurement is stale. Treat the threshold as a tuned parameter with an owner and a re-measurement trigger, not a constant. Second, thresholds only make sense at all with absolute per-state scores. A purely comparative evaluator that ranks siblings gives you order, not level, so there is no scale on which to place a cutoff. ## Depth drift A single global threshold quietly assumes scores mean the same thing everywhere in the tree. They rarely do. Early states are short, generic and underdetermined, and evaluators tend to score them cautiously; deeper states are elaborated and specific, and evaluators tend to score them higher — partly because they look more finished, not because they are more likely to succeed. The effect is that one global cutoff prunes hardest exactly where pruning is most dangerous. Killing a root-level branch discards an entire subtree on the weakest available evidence; killing a deep state near a leaf costs almost nothing. The standard corrections are a depth-dependent cutoff — permissive near the root, strict deeper — or scoring relative to the distribution observed at that depth rather than against a fixed number. ## Confidence, not just score When an evaluation is sampled several times, the spread across samples is information the mean throws away. A state scoring 0.35 with tight agreement is a different object from one scoring 0.35 with samples ranging from 0.1 to 0.7. A cheap and effective policy is to prune confidently-low states and give disputed ones a reprieve, either by expanding them anyway or by paying for extra evaluation samples before deciding. ## Operating the threshold Once set, watch it. The observable signals are the pruning rate — what fraction of generated states are being killed — and end-to-end accuracy. If pruning rate climbs while accuracy sags, either the evaluator has drifted or the incoming problem mix has changed. If almost nothing is being pruned, the cutoff is decorative and you are paying for a tree you are not steering. Also recognise when the threshold is the wrong lever. If no cutoff produces an acceptable saving-to-recall trade, the evaluator does not separate good states from bad ones on this task. Tuning the number harder will not fix that; the evaluator has to improve, or the task does not suit search-based reasoning at all. ## The senior answer in one line Measure recall lost against compute saved on labelled problems, sit below the knee of that curve, keep the cutoff depth-aware, and re-measure whenever the evaluator or the workload changes.