Why is beam search rarely used for open-ended LLM text generation?

## What beam search does Greedy decoding takes the highest-probability token at each step and never reconsiders. That is locally optimal and globally short-sighted: a slightly worse token now can open a much better continuation later. Beam search is the standard remedy. It keeps b partial hypotheses ("beams"). At each step it expands every beam with its top candidates, scores every resulting extension by the accumulated log-probability of the whole sequence, and keeps the best b. At the end you return the highest-scoring complete sequence. With b = 1 it degenerates to greedy decoding; larger b explores more of the space. Because raw log-probability sums penalise long sequences (every extra token adds a negative term), implementations normalise by length, usually with a tunable length-penalty exponent. That normalisation is itself a hint that the objective is a proxy rather than the thing you want. ## Where the objective fails The deep problem is not efficiency, it is the objective. Beam search seeks the highest-likelihood sequence, and for open-ended generation the highest-likelihood sequence is not the best text. Human writing is not maximally probable — it is full of locally surprising choices, and its per-token surprise fluctuates. The maximum-likelihood path, by contrast, is the safe, high-frequency, cliché-dense path, and it degenerates into loops: once a phrase is likely, repeating it stays likely, and the search happily walks that lattice. This observation — that likelihood maximisation produces degenerate text while sampling from the truncated distribution does not — is the argument that put nucleus sampling in every serving stack. So the tasks split cleanly. If there is essentially one right answer and the output is short — translating a sentence, transcribing audio, converting a query to a formal expression — searching for the best sequence is meaningful and beam search earns its keep. If the output is long, open-ended, or valued for variety, the objective is wrong before you consider the cost. ## Where the cost fails Even where the objective is defensible, modern serving makes beam search unattractive. It multiplies work per step. Maintaining b hypotheses means b times the per-step compute and b times the cached attention state, for one returned answer. On a shared inference cluster that is throughput you are spending on a single request. It fights streaming. You cannot safely emit a token to the user while a competing beam could still win, because the prefix you already showed might be pruned. Products where the user watches tokens arrive have no clean way to expose it. It fights the agent loop. Long generations, tool calls that inject external observations mid-sequence, and interleaved reasoning make "the whole sequence" a moving target; the search cannot score across a boundary where an external system contributes content. And it needs its own tuning — beam width and length penalty — which becomes another axis to maintain per task. ## What replaced it The idea that a bit of search improves output did not die; it moved up a level of granularity. Rather than searching over tokens, systems generate several complete candidate responses independently and then choose among them — by agreement, by a verifier, by a scoring model, or by running tests. That is cheaper to reason about, parallelises cleanly across requests, works with streaming (you stream only the winner, or the first one to validate), and composes with tool use. It is also honest about what it costs: n candidates is n times the tokens, visible on the bill. For interviews, the useful summary is historical: beam search is the decoder of the neural machine translation era, still correct for that class of task, and displaced for open-ended generation by truncated sampling below and response-level selection above.