A standard agent loop commits. It reasons, acts, observes, and continues down whatever path that produced. If the path was wrong, it discovers this later and tries to recover from wherever it ended up.
A search agent behaves differently. At each step it generates several candidate actions, evaluates the states they lead to, expands the promising ones, and abandons the rest. When a branch dead-ends it backtracks and tries an alternative rather than trying to salvage the failure.
What changed in 2026
- Spending more compute at inference became a recognised lever. Trading additional inference for accuracy moved from research curiosity to a deliberate architectural choice.
- Verifiers made search practical. A reliable way to score partial states is what search needs, and better verifiers made it work on more tasks — see verifier models.
- Cost discipline tightened. As the multiple became visible on bills, teams got specific about which tasks justified it.
- Hybrid approaches spread. Searching only at the steps that matter, rather than uniformly, became the pragmatic shape.
The cost multiple
This is the first thing to internalise, because it governs everything else.
| Branching factor |
Depth |
Nodes explored |
| 1 (no search) |
5 |
5 |
| 2 |
5 |
up to 31 |
| 3 |
5 |
up to 121 |
| 3 |
8 |
very large |
Every node is at least one model call, frequently two once you count scoring. Pruning keeps this from reaching the theoretical maximum, and the multiple over a single pass remains substantial — commonly an order of magnitude.
Which means the accuracy gain has to be worth roughly ten times the cost. On a task where a single pass succeeds most of the time, it is not. On a task where a single pass frequently fails and failure is expensive, it can be clearly worth it.
Scoring is the hard part
Search without a scoring function is just expensive random exploration. You need to answer, at each partial state: is this going somewhere?
Where that is easy, search works well:
Code. Does it compile? Do the tests pass? Objective, cheap, reliable.
Structured output. Does it validate against the schema?
Formal problems. Is the step legal? Does the proof check?
Constraint satisfaction. Are the constraints still satisfiable?
Where it is hard, search struggles:
Open-ended writing. No principled way to score a partial draft.
Judgement tasks. Scoring requires the same judgement the task needs.
Anything subjective. The scorer inherits the ambiguity.
Using the model itself as the scorer is possible and weakens the whole approach — you are exploring a space guided by the same judgement that generated it, which is the confirmation problem from agent reflection at scale. External, cheap, objective signals are what make search genuinely powerful.
Backtracking and side effects
Search assumes you can abandon a branch and return to an earlier state. That is trivial when the state is a conversation — drop the messages and continue from an earlier point.
It stops being trivial the moment the agent acts on the world. An agent that sent an email, wrote to a database, or called a payment API cannot backtrack; the world does not roll back with the conversation.
Two workable structures. Search over plans, execute once — explore alternatives in reasoning, choose the best complete plan, then execute it with no branching. Or restrict branching to reversible actions and require a single committed path for anything with side effects.
The first is generally cleaner and pairs naturally with a human approval gate on the chosen plan, per human-in-the-loop agents. Where execution must branch, checkpointing becomes a hard requirement — see agent replay and checkpoints.
Common mistakes
- Search without a cheap objective scorer. Expensive wandering.
- Uniform branching at every step. Branch where the decision is genuinely uncertain.
- Backtracking past side effects. The world does not roll back.
- No node budget. The cost multiple becomes unbounded.
- Using the generator as its own scorer. Confirmation bias, amplified by exploration.
- Applying it where single-pass usually works. Ten times the cost for a marginal gain.
FAQ
When is this worth the cost?
When single-pass accuracy is genuinely poor, failure is expensive, and a cheap objective scorer exists. All three, not two of three.
How does it compare to just retrying?
Retrying samples independently and discards what it learned. Search reuses the shared prefix and prunes based on what it observed, which is more efficient than N independent attempts for the same exploration.
Does higher reasoning effort substitute for it?
Often, and more cheaply. Raising effort makes the model think more deeply within one path, which handles many cases search was reached for — try that first, per reasoning effort controls.
What branching factor is sensible?
Small — two or three. The cost grows exponentially in depth, so a wide tree becomes unaffordable within a few steps.
Where to go next
For the scoring component search depends on, read verifier models. For the simpler committed-path pattern, ReAct agents, and for the cheaper first thing to try, reasoning effort controls.