skip to content

What is prompt ensembling, and how does it differ from sampling one prompt many times?

level: middleimportance: should knowfreq 35%

answer

  1. several prompts, one input
  2. vary what is asked
  3. reworded instructions or separate demonstrations
  4. combine step plus a tie rule
  5. output-space versus prompt-space

basics

~20 s

Prompt ensembling runs several deliberately different prompts on one input, such as reworded instructions or separate demonstration sets, then combines their answers. Sampling one prompt repeatedly varies only the decoding, while ensembling varies what the model is asked.

solid answer

~50 s

Prompt ensembling builds a set of prompts for the same task, runs each on the same input, and reduces the answers to one with a combine step such as a majority vote. The prompts differ on purpose: paraphrased instructions, disjoint sets of demonstrations, or demonstrations chosen to cover hard cases. The bet is that wording or example choice causes some errors, so prompts that fail on different inputs can outvote each other. Drawing many samples from one prompt is an **output-space** ensemble: the prompt stays fixed and only decoding randomness varies. The Prompt Report survey files both under ensembling, so it is worth asking which one is meant. Cost scales with the set, since three prompts means three calls per input, and errors every prompt shares survive the vote. Reported gains are specific to their tasks and models, so measure on held-out data.

code

plaintext · 9 lines
plaintext
Prompt A: Assign this support ticket exactly one label from the list.
Prompt B: Which one team from the list should handle this ticket? Reply with the label only.
Prompt C: Read the ticket and name the single category from the list that best describes the main request.

List: billing, account-access, shipping, returns, product-defect, privacy, sales, other
Ticket: I was charged twice and now I cannot log in.

Answers: billing | account-access | billing
Result:  billing (2 of 3)

go deeper

for a junior

Recall the shape: several different prompts, one input, one combine step. Be able to name two things that can vary between prompts, such as wording and demonstrations.

for a middle

Walk through the parts, including the tie rule most people forget, and separate prompt-space variety from output-space sampling in one clear sentence. Run a three-prompt label vote by hand.

for a senior

Say what the method cannot do: shared errors survive, and members must be decent alone. Tie every claimed gain to its task and model, then propose the held-out run that would test it here.

for a principal

Treat it as a spending decision. Every member multiplies calls and upkeep, so the case against simply writing one better prompt has to be made with measured numbers from the team's own traffic.

## What prompt ensembling is **Prompt ensembling** runs several deliberately different prompts on one input and reduces their answers to one. The Prompt Report survey (Schulhoff et al., 2024) describes ensembling as using multiple prompts to solve the same problem and then aggregating the responses, often with a majority vote. It says these techniques reduce the variance of outputs and often improve accuracy, at the cost of more model calls. A **prompt** here means written text: an instruction plus any demonstrations. The **ensemble** means several such prompts. It does not mean several models, and it does not mean several samples drawn from one prompt. ## The parts 1. **Build the set.** Write N prompts for the same task, with the same label list and the same output format. 2. **Run each one on the input.** One call per prompt gives N calls per input. 3. **Make the answers comparable.** A label is ready to count. Free text needs more work. 4. **Combine.** Majority vote is the common default. The set also needs a tie rule, because N prompts can return N different answers. Two practical points shape step 1. Every member should return the same set of labels in the same format, or step 3 becomes a parsing problem. Prompt variants can also break the format: a case study in the Prompt Report made variants by reordering a prompt's demonstrations, found that the reordered versions stopped returning well-structured answers, needed an extra extraction prompt, and ended up scoring worse on its one task. ## What can differ between the prompts | What varies | How it is done | Where it is described | |---|---|---| | Instruction wording | Reword the instruction, keep the meaning | Prompt paraphrasing, listed in the Prompt Report | | Demonstration sets | Give each prompt its own subset of examples | Demonstration ensembling, Khalifa et al. (2023) | | Hard-case demonstrations | Build each new prompt from problems the set solves only sometimes | Boosted prompt ensembles, Pitis et al. (2023) | | Task framing | Rewrite the input into several question forms | Ask Me Anything, Arora et al. (2022) | Two things hold across these methods: - **Each member has to work on its own.** Arora et al. took near-random prompts on two benchmarks and varied their demonstrations. A majority vote changed little, and on one benchmark it scored 2 points below the average prompt. - **Different text does not mean independent errors.** The same paper measured strongly overlapping error sets across prompts, even when the template changed. ## How it differs from sampling Pitis et al. separate an **output-space** ensemble, where one prompt stays fixed and answers differ only through decoding randomness, from a **prompt-space** ensemble, where the prompts themselves differ. Self-consistency is the well-known example of the first kind and is covered with chain of thought. The Prompt Report lists it under the same ensembling heading, so the word alone does not tell you which is meant. The two can stack. DiVeRSe (Li et al., 2022) uses 5 prompts and samples 20 reasoning paths from each, which gives 100 paths per question. A prompt-space ensemble can also run each prompt exactly once, so it does not depend on a non-zero temperature for its variety. ## A small worked case Take an eight-label support-ticket classifier with labels such as `billing`, `account-access` and `shipping`, served by three reworded prompts. In this scenario they disagree on one ticket in ten. For the ticket "I was charged twice and now I cannot log in": - Prompt A answers `billing`. - Prompt B answers `account-access`. - Prompt C answers `billing`. The count is `billing` 2, `account-access` 1, so the result is `billing`, for three calls. When all three return different labels no majority exists, and the tie rule decides. A common choice is the answer of the prompt that scored best on held-out data. ## What it costs and what it does not fix - **Calls** multiply by N, and parallel calls wait for the slowest one. - **Upkeep** grows, because N prompts must be versioned and re-evaluated, not one. - **Shared errors** stay. If every prompt returns the same wrong label, a vote returns it too. - **Evidence is scoped.** The gains above come from particular tasks and from models of 2022 and 2023. A later model may behave differently, so the only number that decides a real system is one measured on its own held-out set.

  • Can a prompt ensemble run at temperature zero?
    Yes. The variety comes from the prompts, so each can be a single greedy call. Whether adding sampling on top helps is a separate measurement. Wang et al. (2022) found output-space sampling to be the key driver in their rationale-augmented ensembles, on their tasks and models, so prompt variety alone may gain less than you expect.
  • Why must each member prompt be decent on its own?
    A vote can cancel noise, but it does not add knowledge the prompts lack. Arora et al. (2022) varied the demonstrations of near-random prompts on two benchmarks and found a minor effect, with one benchmark landing 2 points below the average prompt. They improved the prompt format first, then aggregated.

  • A prompt ensemble is just the same prompt run five times.
  • An ensemble needs several different language models behind it.
  • Rewording a prompt guarantees that its errors become independent.
  • A majority vote always beats the best single prompt in the set.
  • Three short prompts cost about the same as one, so calls do not matter.