Three reworded prompts on an eight-label ticket classifier agree on the same wrong label: why can no vote fix that, and how do you measure it?
answer
- votes flip only disagreements
- unanimous and wrong
- ceiling set by shared errors
- overlap of error sets
- compare with the independent baseline
basics
~20 sA vote can only change inputs where prompts disagree, so a shared wrong label survives any vote or weighting. Measure on a held-out set: disagreement rate, unanimous-wrong rate, and pairwise overlap of error sets against an independent-errors baseline.
solid answer
~40 sWhen all three prompts return one label, every vote goes to it, so any positive weighting still selects it. Unanimous errors therefore set a hard ceiling: if 50 of 1,000 held-out tickets are unanimously wrong, a vote cannot pass 95%, even if it wins every disagreement. They arise because the prompts share a model, a label list and often the same ambiguity, and rewording changes less than it seems. To measure this, score each prompt separately and record the disagreement rate, the unanimous-wrong rate and the **Jaccard index** over each pair's error sets. Arora et al. (2022) measured about 42% when varying demonstrations and 40% when varying the template on three benchmarks, against 25% for independent prompts at 60% accuracy. Report the vote beside its best member, because a small gain can be noise.
code
pseudocode · 15 linesfor each ticket in held_out_set:
answers = run every prompt on ticket, keyed by prompt
if all answers are the same:
unanimous += 1
if that shared answer != ticket.gold:
unanimous_wrong += 1
for each pair (a, b) of prompts:
if answers[a] != ticket.gold and answers[b] != ticket.gold:
both_wrong[a, b] += 1
if answers[a] != ticket.gold or answers[b] != ticket.gold:
either_wrong[a, b] += 1
ceiling = 1 - unanimous_wrong / size of held_out_set
disagreement_rate = 1 - unanimous / size of held_out_set
jaccard[a, b] = both_wrong[a, b] / either_wrong[a, b]go deeper
Recall the simple rule: a vote can only change the result when prompts disagree. If they all give one wrong answer, the vote returns that wrong answer too.
Compute the ceiling from the unanimous-wrong rate, and define the Jaccard index over two prompts' error sets. Show why adding weights makes no difference on a unanimous ticket.
Design the held-out run: per-prompt scores, disagreement, unanimous-wrong rate and overlap against the independence baseline. Then read the results, separating fixable prompt problems from bad gold labels and from noise-level gains.
Make this measurement the entry fee for any ensemble proposal. It costs a few offline calls, and it often shows the money is better spent on the prompt or label definitions than on more production calls.
## Why a vote cannot help A **prompt ensemble** runs several different prompts on one input and combines their answers. A vote counts answers, so if all three prompts return `billing` for a ticket that should be `refund`, then `billing` gets every vote. Giving prompts different positive weights changes nothing, because all the weight still lands on one label. The same holds for mixing per-label probabilities when each prompt ranks the same label first. A vote only has power where members **disagree**. That gives you a useful bound before you spend anything in production: - **Floor on errors:** every unanimous error stays. - **Ceiling on accuracy:** 100% minus the unanimous-wrong rate. - **Realistic result:** somewhere below that ceiling, since the vote will not win every disagreement. ## Why errors are shared These reasons follow from the mechanism, and they are practice rather than measured findings: - The prompts run on **one model**, so they share its knowledge gaps. - They share one **label list**, so a truly ambiguous boundary confuses all of them. - Paraphrases often keep the **same framing**, so they are less different than they look. - Some "errors" are **wrong gold labels**, which every good prompt will miss. Audit these first. ## A worked held-out run These are the scenario's figures for 1,000 labeled tickets, with prompts that disagree on one in ten. | Slice | Tickets | Right after the vote | |---|---:|---:| | Unanimous | 900 | 850 | | Two against one | 80 | 52 | | Three-way split, best prompt decides | 20 | 8 | | Total | 1,000 | 910 | The prompts alone score 90.5%, 89.5% and 88.9%. The vote reaches 91.0%, half a point above the best member, at three times the calls. The ceiling is 95.0%, because 50 unanimous errors stay. Half a point on 1,000 tickets is 5 net tickets, so check it with a paired comparison over the tickets where the two differ before believing it. ## What to measure 1. **Per-prompt accuracy** on one held-out set, so that you know the best member. 2. **Disagreement rate:** the share of inputs without a unanimous answer. 3. **Unanimous-wrong rate:** this sets the ceiling. 4. **Pairwise error overlap.** For prompts A and B, the **Jaccard index** over error sets is the number of inputs both get wrong divided by the number either gets wrong. 5. **An independence baseline.** With error rates a and b, independent errors give ab / (a + b - ab). Compare the measured value with it. In the scenario, A misses 95 tickets, B misses 105, and both miss 70. The index is 70 / 130, about 0.54. Independent prompts at those rates would score about 0.05, so these prompts fail together far more than chance predicts. ## What the research found Arora et al. (2022) measured this directly on three benchmarks with five prompts each, on one open model. - Varying demonstrations gave an average Jaccard index of 42.2%, and varying the template gave 39.9%. - For reference, two independent prompts at 60% accuracy would score 25%. - Their simple majority vote, on two of those benchmarks with varied demonstrations, changed little and landed 2 points below the average prompt on one. Their result is limited to those benchmarks and one open 6-billion-parameter model. It does show that **changing the template did not remove the overlap**, so rewording alone is not good evidence of independence. ## Disagreement and overlap are different numbers The two are easy to conflate, and they answer different questions. **Disagreement** counts inputs where answers differ, including inputs where one prompt is right. **Overlap** looks only at failures and asks how often they coincide. Two prompts can disagree on 10% of tickets and still share most of their errors, as they do above, because about half of each prompt's misses fall in the unanimous slice. A team that reports only the disagreement rate will conclude the prompts are diverse, while the overlap says their failures are not. ## What to do with the result - **High unanimous-wrong rate:** no combine step helps. Fix the prompt, the label definitions or the gold labels. - **High disagreement, low overlap:** a vote has room to work, so test it. - **High disagreement caused by one weak prompt:** remove or repair that prompt, because the vote may be spending its calls on noise. - **Try changing something larger:** different demonstrations and task framing, then re-measure the overlap. A larger change may also lower each member's accuracy, so track both numbers.
- Does a high disagreement rate mean a vote will help?No. Disagreement is necessary but not sufficient. One weak prompt can create most of it: the other two already agree, so its vote changes nothing and still costs a call. Check accuracy on the disagreement slice and compare the vote with its best member on the same tickets.
- Could a weighted vote rescue unanimous errors?A weighted vote cannot, because all positive weight lands on the one label returned. Fixing such errors needs a signal outside those votes: a better prompt, corrected label definitions, more context in the input, or a separate check. Weighting only changes how disagreements resolve.
- Why is the unanimous-wrong rate worth reporting separately from overall accuracy?Overall accuracy mixes errors a vote could fix with errors it never can. The unanimous-wrong rate is the part no combine step touches, so it gives a ceiling before any production spend. If that ceiling sits close to the best prompt's score, the ensemble has almost no room to pay for itself.
Three witnesses who all read the same mistaken newspaper report will agree with each other. Their agreement feels like confirmation, but it adds no independent evidence, and asking one more reader of that paper would not correct the story.
- With more voters, any error eventually gets outvoted.
- Three differently worded prompts have independent errors by construction.
- Weighting the most accurate prompt higher can fix unanimous mistakes.
- When all prompts agree, the answer can be assumed correct.
- A half-point gain on 1,000 tickets proves the vote works.