When does spending more test-time compute stop paying off for a task?

## The two curves When people say "more thinking helps", they are describing one regime and generalizing it to all traffic. Separate them. **The hard-problem curve.** On problems with genuine sequential structure — a puzzle requiring induction over examples, a schedule with interacting constraints, a proof — accuracy really does rise as you spend more inference-time compute on identical weights. It rises **roughly logarithmically**: the first big increment in thinking buys a lot, the next buys less, and eventually the line goes flat. There is a knee, and it is task-specific. **The easy-problem curve.** On lookups, classifications and one-line answers, the curve is flat from the origin. Spending here is not a smaller gain; it is no gain. A route that answers "what timezone is Denver in" and burns three thousand thinking tokens doing it has produced a latency and cost regression with zero accuracy benefit — and occasionally a negative one, when the model talks itself out of a correct first instinct. Real traffic is a mixture of both. A single global effort setting is therefore always wrong for part of your traffic; the only question is which part you have chosen to be wrong about. ## What actually determines the payoff The useful test is: **what is the binding constraint on this answer?** Spending pays when the constraint is *computation the model can perform*. If the model has everything it needs and simply has to enumerate, evaluate, check and backtrack, more tokens are more of that work. Spending does not pay when the constraint is anything else: - **Missing information.** No amount of deliberation recovers a fact outside the model's knowledge or a document not in context. The fix is retrieval or a tool call, not depth. - **Ambiguous specification.** If the request admits several readings, thinking elaborates one reading confidently. The fix is a clarification or a better prompt. - **Unverifiable quality.** If nobody — not the model, not a checker — can tell a better answer from a worse one, the model has no gradient to climb during deliberation. Extra tokens produce polish, not correctness. A closely related asymmetry: deliberation pays most where the model can **check itself**. Arithmetic can be re-derived; a constraint can be re-tested; code can be traced. Where a self-check is possible, extra thinking finds errors. Where the answer is a judgement call, extra thinking mostly generates justification for whatever it first believed. ## Over-reasoning as a production defect Treat this as a real regression class, not a curiosity. Symptoms: p95 latency multiplying after thinking is enabled by default; a bill dominated by output tokens on routes whose answers are short; users abandoning a surface that used to feel instant. The cause is a depth policy applied uniformly to a mixed workload. Diagnosis is straightforward if you instrumented for it: log thinking tokens per response, bucket by route and by request class, and look at the distribution rather than the mean. The pathology is visible as a fat tail on routes whose answers are trivially short — high spend, low output, no accuracy difference against a lower-effort baseline. The remedies are ordinary engineering: drop the depth setting or disable thinking on shallow routes; keep depth for the routes where your eval shows it moves accuracy; make depth a per-request-class decision rather than a deployment-wide one. ## How to find the knee, honestly Published curves are about published benchmarks. Yours will differ, so measure: 1. Build an eval set per request class from real traffic, including the hard tail rather than a convenient sample. 2. Run each class at each available depth setting, several times per item, since output varies run to run. 3. Plot accuracy against actual measured cost and latency — not against the setting's name, which is not a unit and does not transfer across model families. 4. Find where the accuracy line flattens. Set the default there. 5. Re-run this after any model change. A new model can move both the accuracy and the cost at every level, and settings tuned on the old one silently become wrong. The discipline that separates a senior answer from a principal one is step 3: reasoning about accuracy **per dollar and per second**, not accuracy alone. Any depth setting improves accuracy somewhere; the question is what you paid and whether that beat spending the same budget on retrieval, better context, or a verifier. ## Where to spend the budget you have With a fixed budget, put depth where two conditions hold together: **errors are expensive**, and **the task is checkable**. A root-cause analysis on a live incident qualifies on both counts — a wrong conclusion costs an outage extension, and the reasoning can be validated against the evidence. A tag-suggestion feature qualifies on neither. And hold the alternatives in view. More thinking is one lever among several. Giving the model the right document, tightening an ambiguous instruction, or adding a real verifier that catches errors after the fact will often beat a depth increase at the same cost — and unlike depth, those do not have a flat region waiting a doubling away.