Greedy decoding picks the single most likely next token at each step and never reconsiders. That is short-sighted — a token that looks best now may lead somewhere poor, and a slightly worse choice might have led somewhere much better.
Beam search addresses that by keeping several candidate sequences alive simultaneously, extending each, and retaining the highest-scoring ones. At the end it returns the sequence with the best overall probability.
It is a better search. It produces worse text for most of what people use language models for, and understanding why is genuinely illuminating.
What changed in 2026
- Sampling remained dominant for open-ended generation. Nothing displaced it, and the reasons became better understood.
- Beam search stayed standard for constrained tasks. Translation and short structured generation continued to use it.
- Structured outputs absorbed some of its use. Schema-constrained decoding handled cases where beam search was previously used for format reliability.
- Reasoning models shifted the conversation. Search moved up a level, from token sequences to reasoning paths — see tree search agents.
The likelihood trap
The finding that explains everything: the most probable sequence is not the best sequence for open-ended text.
High-probability text is text that is unsurprising. Given a prompt, the highest-probability continuation tends toward the generic, the repetitive, and the safe — common phrasings, hedged statements, and structures the model has seen constantly.
Beam search optimises for exactly that, and it does so more effectively than greedy decoding, which is why its output is frequently worse rather than better. A wider beam finds higher-probability sequences, and higher-probability sequences are blander. Increasing beam width past a small number reliably degrades perceived quality on open-ended tasks.
It also has a documented tendency toward repetition, because repeating a phrase already used is high-probability by construction.
Human-preferred text contains some surprise. It makes specific choices, uses less common phrasings, and takes positions. Sampling introduces exactly that variation, which is why it produces text people prefer despite being lower-probability.
|
Greedy |
Beam search |
Sampling |
| Optimises |
Local probability |
Sequence probability |
Neither; samples |
| Output character |
Repetitive |
Blander, more repetitive |
Varied |
| Cost |
1x |
Proportional to width |
1x |
| Best for |
Short deterministic tasks |
Constrained tasks |
Open-ended text |
| Reproducible |
Nearly |
Nearly |
No |
Where it still works
Beam search is not obsolete; it suits a specific shape of task.
Machine translation. There is a correct answer, or a small set of correct answers, and the goal is to find the most probable rendering. Blandness is not a defect when fidelity is the objective.
Short constrained generation. A structured field, a classification label, a formatted identifier — cases where one output is right and creativity is unwanted.
Anything scored by likelihood. If your evaluation metric rewards probable output, beam search will do well on it, which is worth noting as a reason to be sceptical of such metrics.
The common thread: a well-defined correct answer, short output, and no value in variation. Chat, creative writing, explanation, and summarisation all fail those conditions, which is why chat models sample.
Cost
Beam search generates several sequences simultaneously, so cost scales roughly with beam width. A width of five costs about five times a single sequence.
For short constrained outputs that is affordable. For long-form generation it multiplies an already-expensive operation to produce output people prefer less, which is the worst combination available.
Where you want to spend more compute for better output on open-ended tasks, the productive direction is search over reasoning paths with a verifier, not search over token sequences by probability — see verifier models.
Common mistakes
- Beam search for chat. Blander output at higher cost.
- Wider beams expecting better quality. Reliably worse on open-ended text.
- Confusing it with sampling parameters. Different mechanism entirely — see sampling parameters.
- Using it for format reliability. Structured outputs do this properly.
- Evaluating with likelihood-based metrics. They favour exactly what humans do not.
- Assuming higher probability means higher quality. The core error.
FAQ
Why do translation systems still use it?
Because translation has a correct answer and no value in creative variation. The properties that make beam search bad for chat make it appropriate there.
Is greedy decoding beam search with width one?
Effectively yes — one candidate, always the most likely token. Which is why greedy output shares beam search's repetitiveness in milder form.
Should I ever use it with a chat model?
For short constrained outputs where one answer is correct, occasionally. Structured outputs usually serve that purpose better and more cheaply — see structured outputs.
What about spending more compute for better output?
Search over reasoning paths, scored by a verifier, rather than over token sequences by probability. That is where inference-time compute genuinely buys quality — see tree search agents.
Where to go next
For the parameters that govern sampling, read sampling parameters. For guaranteed output shape, structured outputs, and for productive inference-time search, tree search agents.