In Tree of Thought, how do you design the propose and evaluate prompt templates?
answer
- Two roles, two output contracts
- Hot proposer, cold evaluator
- Path rendered by one function
- Constant blocks first, variable last
- Parse failure is not a low score
basics
~20 sWrite two separate templates with different jobs: a proposer that receives the serialized path and returns candidate next steps as free text, and an evaluator that receives one candidate state and returns a strictly parseable verdict. Keep their shared prefix byte-identical.
solid answer
~50 sTreat them as two distinct roles with two distinct output contracts. The **proposer** gets the task statement, the rules, and the path so far rendered as an ordered list, and is told to emit only the next step — no commentary, no restating the problem — so the parse is trivial. The **evaluator** gets the same task statement and one candidate state, and is told to emit only a verdict token in a fixed format, which you parse strictly and retry on failure rather than regex-scraping prose. Sampling differs by role: the proposer runs hot (temperature around 1.0) because you want diverse siblings, while the evaluator runs at temperature 0 so the same state scores the same way twice. Two further habits pay for themselves: keep the shared prefix — task plus rules — identical across every call in an expansion so providers can serve it from prompt cache, and do not show the evaluator which model or branch produced a candidate, or you invite self-preference bias.
code
python · 21 linesTASK = "Plan a zero-downtime split of the orders table into orders and order_items."
RULES = "Each step is one reversible DDL or backfill action. No step may lock orders."
PROPOSE = (
"{task}\n{rules}\n\n"
"Steps so far:\n{path}\n\n"
"Give the next step. Output only the step text."
)
EVALUATE = (
"{task}\n{rules}\n\n"
"Candidate plan:\n{path}\n\n"
"Reply with one integer from 0 to 10 and nothing else."
)
def render(template, path):
body = "\n".join(f"{i + 1}. {s}" for i, s in enumerate(path)) or "(none)"
return template.format(task=TASK, rules=RULES, path=body)
print(render(PROPOSE, ["Create empty order_items table"]))
print(render(EVALUATE, ["Create empty order_items table", "Backfill in batches"]))go deeper
Know that the loop uses two different prompts, one asking for a next step and one asking for a score, and that the score prompt must return something your code can parse.
Explain the output contract, the hot-proposer and cold-evaluator temperature split, and why the path is re-rendered into every prompt by a single function.
Demonstrate the operational habits: a stable cached prefix, strict parse-or-retry, provenance-blind evaluation to avoid self-preference, and compact state because the path cost multiplies with branching.
Own the templates as versioned interfaces — changing the shared prefix invalidates caches and makes historical runs incomparable, so template versioning belongs in the run log and the rollout plan.
## Why two templates, not one clever one The temptation is a single prompt that says 'propose three next steps and rate each one'. It is cheaper by one round trip and worse in almost every other way. Ratings produced in the same pass as the proposals are conditioned on those proposals — the model is grading its own fresh output inside one autoregressive stream, and it grades generously. You also lose the ability to sample the two roles differently, to route them to different models, to retry one without redoing the other, and to swap the evaluator for a non-model checker later. Two templates is the shape that survives contact with production. ## The proposer template Structure it as four blocks in a fixed order: task statement, rules and constraints, the serialized path so far, and the instruction. The first two blocks are constant for the whole run — that constancy is what makes caching possible. The path block is the only thing that varies per node, and it should be rendered by one function so every call formats it identically: an ordered list of the steps taken, or `(none)` at the root. The instruction should ask for exactly one artefact per call and nothing else. 'Output only the step text' is worth more than any amount of persona framing, because the orchestrator has to parse the result into a node. If you ask for several candidates in one response, you have to parse a list, and a malformed list costs you the whole call; if you ask for one candidate and sample the call several times, each failure costs you one child. Both are defensible; be able to say which you chose and why. ## The evaluator template Same first two blocks — byte-identical to the proposer's, which is the point — then one candidate state, then an output contract. The contract is the whole game. Say exactly what may be emitted and nothing else, parse it with a strict parser, and treat a parse failure as a retryable error rather than something to salvage with a regex. A verdict you cannot parse is not a low score; conflating the two silently biases the search against whichever states happened to trigger chatty responses. The evaluator should be blind to provenance. Do not tell it which branch, which model, or which sampling temperature produced the candidate, and do not include the proposer's own hedging in the state you show it. Models exhibit self-preference — they rate text that looks like their own output higher — and leaked provenance turns a scoring function into a popularity contest between prompts. ## Serializing state back into the prompt Everything the model knows about a node is what you render. Three properties matter: **Compactness.** The path is re-sent on every child expansion and every scoring call, so its token cost is multiplied by the branching factor at every depth. A verbose state representation is the most common reason a tree run costs ten times the estimate. **Determinism.** Render through one function. If two call sites format the same path differently — one with numbering, one without — you break prefix caching and you make two logically identical calls incomparable in your logs. **Completeness.** The rendering must contain everything needed to continue; anything you keep only in your own objects (a constraint discovered at depth 2, say) is invisible to the model at depth 5. ## Sampling parameters as part of the template Treat temperature as part of the template contract, not an afterthought. The proposer at temperature 1.0 (or with top-p loosened) is what makes siblings genuinely different rather than three rewordings of one idea. The evaluator at temperature 0 is what makes scores comparable across nodes: if the same state can score differently on two calls, your pruning decisions are partly noise, and a run is not reproducible even with everything else logged. Some teams go further and average several evaluator samples, but that is a scoring-design decision with its own cost; the baseline discipline is simply cold, deterministic scoring. ## Prefix caching, concretely Providers generally serve a repeated prompt prefix from cache at a large discount on input tokens. In a tree run, the task statement and rules repeat across every single call — often dozens per expansion — so making that prefix stable and putting it first is one of the highest-leverage things you can do for cost. It has a design consequence: anything variable (the node's path, a timestamp, a node id) must go *after* the constant block, never interleaved into it. ## What interviewers listen for They want role separation, a machine-parseable evaluator contract, a single deterministic state renderer, and a reason for each temperature. Candidates who describe one mega-prompt, or who scrape scores out of prose with a regex, have usually not run this loop against a real workload.
- Why not ask for the candidates and their scores in one call to halve the round trips?Because the scores are then generated conditioned on the proposals in the same pass, which makes them systematically generous and correlated. You also lose per-role sampling, per-role retries, the option to route roles to different models, and the ability to drop in a non-model checker later. The saved latency rarely covers a scoring function you cannot trust.
- The evaluator sometimes replies with a number plus a sentence of justification. How should the orchestrator handle that?Treat it as a contract violation: fail the parse and retry the call, optionally with a stricter reminder. Do not regex the first number out of the prose — that silently accepts drifting output and hides the fact that your contract is eroding. If you actually want a rationale, put it in the contract explicitly as a second field, so parsing stays total rather than best-effort.
- What breaks if the path renderer is duplicated at two call sites?Two things. Formatting drift breaks the shared-prefix assumption and quietly loses your cache discount, and it makes logically identical calls non-comparable in logs and replays, so a diff between two runs shows spurious differences. One renderer, called everywhere, is the cheap fix.
saying these in an interview costs you the question
- Uses one prompt that proposes and scores in the same pass
- Scrapes evaluator scores out of prose with a regex
- Runs proposer and evaluator at the same temperature
- Puts variable text before the constant task block
- Tells the evaluator which branch or model produced a candidate