In Tree of Thought, when does an external verifier beat an LLM evaluator?

## The evaluator slot is pluggable Because a tree run separates proposal from evaluation, whatever fills the evaluator slot only needs to accept a serialized state and return something the control policy can compare. Nothing requires it to be a model. In any domain with a checker, using that checker is the strongest available upgrade to a tree search, because it replaces a judgement with a fact. The usual candidates: a compiler or type checker for code steps; a unit or property test suite for a completed function; a schema or config linter for infrastructure changes; a query planner run against a shadow copy of a schema for migration steps; a constraint or LP solver for scheduling; a simulator for control policies; a parser for anything with a grammar. ## What a verifier buys **Soundness on rejection.** When a compiler says the code does not compile, that is not an opinion. A model evaluator can be talked into approving something malformed, especially when the malformed thing reads fluently. **Determinism.** The same state produces the same verdict every time, which makes the search reproducible and makes a regression attributable to a proposal change rather than to scoring noise. **Independence.** The verifier has no stake in which model wrote the candidate, so the self-preference bias that afflicts model-judged pipelines disappears from the pruning decision. **Cost profile.** Running a linter or planning a query is typically far cheaper and faster than an extra model call, so verifier-first pruning can shrink the frontier before you spend model tokens on it. ## What a verifier cannot do **Rank.** Most checkers are binary. Once ten candidate migration steps all plan successfully, the planner has nothing more to say about which is the better next move — and ranking is exactly what a search needs to guide expansion. This is the reason 'just use tests instead of a model' is not a complete answer. **Judge partial states.** A verifier usually wants a well-formed artefact. Halfway through a plan there may be nothing to compile and no query to plan, so the evaluator slot is empty precisely where the search needs guidance most. Some domains work around this with a relaxed check (does the partial plan still satisfy the invariants so far?), but the relaxation is itself engineering work. **Cover intent.** Passing tests, a clean lint and a good query plan together do not mean the plan achieves the goal. A verifier constrains the space; it does not confirm the destination. ## The hybrid that people actually ship Run the verifier first as a **hard filter**. Children that fail are pruned immediately and cheaply, with the failure text recorded. Children that pass go to a model evaluator that only has to answer the softer question — which of these valid options looks most promising — which is a question models are comparatively good at, and now they are only asked about states that are known well-formed. The compute saving is real: you stop paying model tokens to score candidates a parser could have rejected. The second half of the hybrid is the feedback path. A verifier failure carries a *reason* — a compiler error, a failing assertion, an index the planner ignored. Feeding that text into the proposer when it retries that region of the tree converts a dead branch into information. This is not the same as a general critique loop: the signal is a fact from a tool, and it is scoped to one node's regeneration. ## The operational catches **Side effects.** Running real artefacts is the whole point, and it is also the danger. Verification must happen against a disposable copy — a shadow schema, a scratch branch, an ephemeral container — with no path to production data. Never verify a destructive candidate by executing it somewhere that matters. **Latency and concurrency.** A verifier that takes seconds per call may be slower than the model it replaced, and running dozens concurrently may need process or container isolation. This can quietly become the bottleneck of a level. **Flakiness.** A flaky test suite in the evaluator slot is worse than a noisy model, because you will trust it. Verifier flakiness must be detected and treated like an unscored node, not like a rejection. **Partiality bias.** If the verifier can only judge a subset of states, and unjudgeable states default to 'pass', the search drifts toward exactly the candidates nobody checked. Make the unjudgeable case an explicit, logged category. ## The interview answer Name a concrete verifier for a concrete domain, say what it makes sound, admit that it filters rather than ranks, and describe the hybrid plus the sandbox. Candidates who claim a verifier replaces the model evaluator entirely have usually not watched a frontier of ten equally-valid candidates with no ordering signal.