In Tree of Thought, when do you sample candidate thoughts independently instead of proposing them in one call?

## What thought generation has to produce Tree of Thought turns reasoning into a search. At every state in the tree — a partial solution so far — something has to produce several *candidate next steps*, called thoughts. Those candidates are the branches. If generation returns one usable option, the search degenerates into a single line of reasoning with extra overhead; if it returns k genuinely different options, the rest of the machinery has something to work with. Generation is therefore the part of the loop that decides what the search is even allowed to find. There are two standard strategies for producing those k candidates, and the choice is the first design decision you make. ## Independent sampling (i.i.d. draws) Here you build one generation prompt — the task, the current partial state, an instruction to produce *one* next step — and you run it k times. Each run is an independent draw from the model's distribution over next steps. Nothing in the fourth call knows what the first three produced. Properties worth stating in an interview: - **Parallelism.** The k calls have no ordering dependency, so wall-clock latency is roughly one call, not k. That matters when a tree of depth d and breadth b already costs on the order of b·d generations. - **No positional bias.** Every candidate is written as if it were the only one, so none of them is a grudging afterthought. - **Diversity is a property of the space, not of the prompt.** In a rich, open-ended space — the next paragraph of a short story that must land a plan point, a design approach, a hypothesis for an outage — the distribution is broad enough that independent draws land in different places on their own. - **No duplicate control.** Nothing prevents two draws from being the same or near-identical, because neither one can see the other. ## Sequential proposal (enumerate in one call) Here a single call is asked to produce all k candidates at once: "list four possible next steps." Because the model writes them into one continuation, candidate three is conditioned on candidates one and two. You can also instruct the model directly — "each option must differ from the previous ones" — and that instruction actually has something to bind to. Properties: - **Built-in de-duplication.** The conditioning is the whole point. In a constrained space this is the difference between four options and one option repeated four times. - **Cheaper and simpler.** One request, one response, one parse; the shared prefix (task and state) is written once rather than k times. - **Quality decay down the list.** Models tend to spend their strongest options first and then pad. The fifth item is often weaker or more contrived than the first, which distorts what the search sees. - **Serial by construction.** Latency is one long generation, and a truncation or a malformed list costs you every candidate at once. ## Choosing between them The deciding question is: *how big is the space of reasonable next steps, and would independent draws collapse?* - **Small, constrained, enumerable-ish spaces → propose.** Filling a crossword slot where only a handful of words fit the clue and the crossing letters; picking among a short list of legal operations. Independent draws here waste calls rediscovering the top-probability answer. - **Large, open-ended, generative spaces → sample.** Continuing a story under a plan constraint; drafting alternative outlines. Independent draws are already varied, and you avoid the list-padding artifact. A third consideration is *how many* candidates you want. For k of two or three, proposal's decay is mild and its cheapness wins. For larger k, sampling scales better because you never ask one continuation to carry the whole set. ## Hybrids The two strategies compose. A common pattern is to run the propose call a small number of times independently and take the union — each call internally avoids duplicates, and the independent runs cover different regions — then drop near-identical candidates before expanding. Another is to sample independently and then feed the batch back with an instruction to add one option unlike any of them, filling gaps in coverage. ## What this is not Generation only decides *what the options are*. Deciding which candidates look promising, and which branch to expand next, are separate stages with their own designs; conflating them is the most common muddle in an answer. Keep the claim narrow: sampling versus proposing is a statement about how candidates are drawn and whether they can see each other.