When does ensembling independent LLM samples stop improving accuracy?

## The mechanism, stated properly Ensembling runs the same task N times and collapses the results: majority vote for a discrete label, median or trimmed mean for a number, a judge or an aggregator agent for free text. The intuition borrowed from classical statistics is that independent errors point in different directions and partly cancel, so the aggregate is more accurate than a typical single sample. That intuition rests on two assumptions, and both are load-bearing. **Each sample must be better than chance.** If the per-sample probability of being right is below one half on a binary decision, majority voting drives accuracy down, not up, and more samples make it worse. The ensemble amplifies whatever the base rate is; it does not create competence. **Errors must be at least partly independent.** This is the assumption that fails in practice with language models. Two samples from the same model on the same prompt are not two independent observers. They share weights, training data, and — crucially — the framing of the prompt and the contents of the retrieved context. When the model has a systematic bias, every sample carries it. ## The demand-forecast case Consider a system that samples five demand forecasts for next quarter and aggregates them by median. The median is a genuinely good aggregator for numeric output: it is robust to one run that produces an absurd figure, which a mean is not. So the design is sound as far as it goes. Now suppose the retrieved context omitted a promotional campaign. All five samples read the same context, all five miss the same driver, and all five come back low by roughly the same amount. The five figures cluster tightly. Downstream, the tightness is reported as agreement and therefore as confidence — and the ensemble has manufactured false certainty about a number that is systematically wrong. Adding five more samples narrows the cluster further and changes nothing about the error. The lesson generalises: **spread across samples measures the model's variance, never its bias.** An ensemble can only fix variance. ## Where diversity actually comes from Ranked from weakest to strongest source of decorrelation: - **Temperature and seed.** The cheapest lever and the shallowest. It perturbs the sampling path, so it catches errors that are artefacts of one unlucky decoding, but leaves every systematic prior intact. - **Prompt and framing variants.** Asking the same question three different ways — different decomposition, different output format, different emphasis — decorrelates more, because framing drives a large share of model behaviour. - **Different context or tool paths.** Letting each run retrieve independently, or approach the problem with different tools, decorrelates the inputs as well as the reasoning. This is where an omitted-driver error can actually be caught. - **Different base models, ideally from different providers.** The strongest available decorrelation, because the shared training data and shared architecture bias is broken. It is also the most operationally annoying: different formats, different failure modes, different costs. If you take one thing into an interview: diversity has to be designed in, and the cheap lever is the weak one. ## Choosing the aggregator The aggregation rule should match the output type. Majority vote for discrete labels. Median or trimmed mean for numbers, because both resist a single outlier run. For free-form text there is no arithmetic aggregator, so you either pick a representative sample by similarity to the others, or hand the candidates to a judge or aggregator agent — which introduces a new component with its own failure modes. And where a checker exists — the code runs, the constraint holds, the total reconciles — filter with the checker before aggregating anything, because verification beats voting outright. ## Measuring instead of assuming Three numbers make this concrete on your own data: - **Per-sample accuracy**, so you know whether you are above the threshold where voting helps at all. - **Aggregate accuracy at N**, plotted as N grows. The curve flattens; find where, and stop there rather than at a round number. - **Best-of-N (oracle) accuracy versus the aggregate.** A large gap means the right answer was in the pool and the aggregation rule failed to select it — a selection problem, fixable with a verifier or a better judge. A small gap means the right answer was rarely generated at all — a capability or grounding problem, and no amount of sampling will fix it. That last diagnostic is the strongest thing you can say on this topic, because it tells you whether to invest in aggregation or abandon it. ## Cost Ensembling multiplies cost by N with no compounding benefit once the curve flattens. Unlike sequential debate, the samples parallelise, so latency is roughly one call plus aggregation — which is why ensembling is often the cheaper accuracy lever in wall-clock terms even when it is the more expensive one in tokens. Selective application, where only low-confidence or high-stakes requests get the full N, keeps the bill proportional to the value at risk.