A debate ensemble costs 10x tokens for 4 points of accuracy — do you ship it?

## Get the multiplier right first People quote agent counts and forget the shape of the spend. Three debaters over three rounds is nine generations, plus a judge call, so ten. But each round's prompt carries the accumulated transcript, so input tokens grow superlinearly across rounds while output tokens grow linearly. On long-context tasks the input side dominates. A useful discipline is to measure the ratio empirically against your single-call baseline rather than reasoning from the agent count — the observed premium on research-style orchestration in 2026 practice runs around fifteen times a plain chat interaction, which is well above what a naive multiplication suggests. Latency has a different shape from cost. Independent samples fan out and complete in roughly one call's time. Debate rounds are strictly sequential: round two cannot start until round one finishes for every participant, so wall-clock time scales with round count and each round runs at the speed of the slowest debater. A design that is affordable can still be unshippable behind an interactive latency budget. ## Is the four points real? Before the economics, interrogate the number. **Statistical reality.** A four-point difference on a two-hundred-example eval set is frequently inside the noise band. Compute an interval, or bootstrap it. Run the comparison more than once, since both arms are stochastic. A gain that vanishes on a second run was never there. **Construct validity.** A gain measured on a public benchmark may not transfer to your product, because benchmark items are cleaner, shorter and better specified than real traffic, and popular benchmarks suffer contamination. Reproduce the comparison on cases sampled from your own production traces. **Where the gain lives.** Slice it. Very often the entire improvement sits in one hard segment — ambiguous inputs, long documents, an underrepresented category — and the easy majority of traffic is unchanged. That finding does not kill the technique; it changes the deployment from global to selective, which changes the economics completely. ## Price the error, not the accuracy Accuracy points are not fungible. What matters is the expected cost of the errors the extra points remove, against the cost of the tokens. Write it out for the concrete case: requests per day, the fraction that hit the hard slice, the four-point delta on that slice, and the cost of one wrong outcome — a bad clinical triage, a wrongly upheld moderation appeal, a mispriced quote, a merged defect. When one error costs more than thousands of extra generations, the decision is obvious and the debate is cheap. When errors are recoverable, cheap to correct downstream, or reviewed by a human anyway, a ten-times multiplier for four points is a poor trade and the honest answer is no. Asymmetry matters as much as magnitude. If false positives and false negatives cost wildly different amounts, check which side the four points came from. A gain concentrated on the cheap error class is worth much less than the headline suggests. ## The opportunity-cost test The comparison that separates a principal answer from a senior one is not debate versus a single call — it is debate versus everything else you could buy with the same budget. - A stronger model on a single call, at similar total spend. - Better grounding: improved retrieval, cleaner context, the right documents present at all. - A deterministic verifier that catches the same errors for a fraction of a generation. - One extra reasoning pass rather than an entire second and third agent. This is not rhetorical. Results published through 2026 repeatedly show single-agent configurations matching or beating multi-agent ones once token budgets are equalised, and the additional agents earning their place mainly through context isolation rather than through extra deliberation. Any proposal to ship debate should include the matched-budget single-agent arm; without it the comparison is unfair to the baseline. ## Selective application The usual right answer is not "ship it" or "drop it" but "ship it for part of the traffic". Gate the expensive path on stakes — value at risk, irreversibility, regulatory exposure — or on a cheap uncertainty signal from the first pass. Two properties make this work: the gate must be much cheaper than the path it guards, and the fraction escalated must be measured and bounded, or a distribution shift quietly converts your selective spend into global spend. Set a budget cap and alert on the escalation rate. ## The operational bill Tokens are not the whole cost. A debate ensemble adds failure surfaces: a debater that times out, a judge that returns malformed output, a round that exceeds the context window. Attribution gets harder — when the final answer is wrong, deciding which agent caused it is an open problem, with step-level attribution accuracy on published benchmarks still poor even with full traces. Debugging, tracing and on-call load all rise. Fold that into the decision; it is the part that usually goes unbudgeted. ## How to answer in the room Say: I would validate that the four points is real and find which slice it lives in, price the errors it removes against the multiplier, insist on a matched-budget single-agent comparison, check the sequential-latency implication against the SLO, and then most likely ship it behind a stakes-based gate with a bounded escalation rate — rather than as the default path.