A Tree of Thought turn costs thirty model calls under a two-second chat SLA — how do you fix it?

## The cost model you should be able to state Tree of Thought's spend is structural, not incidental. For a tree with branching factor b explored to depth d, the number of expanded nodes grows with b and d, and **each node costs at least one generation call plus one evaluation call** (more if you sample several evaluations per state to reduce judgement noise). So total calls land in the neighbourhood of nodes times calls-per-node, and tokens are worse than that because each expansion re-sends accumulated context. Two consequences matter operationally: - **Cost scales with the whole tree.** Thirty calls per turn against one for a chain is a thirty-times bill for the same user-visible answer, before token growth. - **Latency scales with depth, not with total calls** — but only if you actually parallelise. Siblings at the same level are independent and can be issued concurrently; levels are strictly serial because you cannot expand a node before you have scored its parent. Best-case wall clock is roughly d sequential round trips plus evaluation time. ## Applying it to a two-second chat SLA Even a shallow tree — say depth three with a scoring round at each level — costs six sequential model round trips. On typical interactive latencies that alone eats the budget, and any node that needs more than one evaluation sample makes it worse. So the first honest answer in an interview is: **the SLA and the architecture are in conflict, and one of them has to move.** ## The remedies, in order **1. Question whether the turn is search-shaped at all.** Most support-chat turns are retrieval, a policy lookup or a status check. Those have no scoreable partial state and no dead end, so a tree buys nothing and costs thirty times as much. Send them down a single chain and reserve any search for the small tail of genuinely combinatorial requests. **2. Shrink the tree.** Cut the branching factor, cap depth, and prune harder on the evaluator's score. Quality usually degrades gracefully here, which makes it the cheapest lever to tune against an eval set. **3. Parallelise within a level.** Issue all sibling generations concurrently, then all evaluations concurrently. This does nothing for cost but converts a thirty-call serial chain into a handful of sequential rounds. **4. Make evaluation cheap.** Scoring calls are often the majority of the tree's calls. A programmatic verifier — a regex, a schema check, a constraint check, a test run — is both faster and more reliable than a model judging its own partial output. Where you need a model, a smaller one is usually adequate for ranking. **5. Get it off the interactive path.** If the work is genuinely search-shaped and valuable, run it asynchronously: acknowledge in the chat, do the search in a background job, deliver the result when it lands. Many products discover the search-shaped requests were never really conversational. **6. Consider a weaker but bounded alternative.** Sampling a handful of independent chains and taking the majority answer costs a fixed, parallelisable number of calls with no serial depth at all. It captures part of the benefit of exploration where a full evaluated tree cannot fit the budget. ## What to measure Do not defend the architecture on anecdote. Track cost per resolved turn and p95 turn latency alongside task success, and compare a chain, bounded sampling, and the tree on the same eval set. The decision rule is quality gain per unit of cost and latency, not raw accuracy. A five-point accuracy gain that turns a one-second turn into an eight-second one is usually a regression in a chat product and a bargain in an offline batch pipeline — the same technique, judged by the surface it runs on. ## The 2026 framing Before building tree orchestration for a latency-sensitive surface, benchmark the same model with a larger thinking budget. Reasoning models spend test-time compute on internal exploration in one request, which is one round trip rather than d of them, and providers expose effort or thinking-budget controls to tune that spend. That comparison frequently decides the question without any orchestration code, and it is the comparison a senior interviewer is listening for.