Chain of thought prompting is the deceptively simple idea that a model produces better answers when it works through the problem in steps rather than blurting out a conclusion. Ask a model to "think step by step" and, on the right kind of problem, accuracy can jump noticeably. But the technique is widely over-applied, and the arrival of dedicated reasoning models has changed when you actually need it.
What changed in 2026
- Reasoning models absorbed the technique. Many models now perform extended internal reasoning by default, so explicitly prompting "think step by step" adds far less than it did on earlier models.
- Faithfulness got scrutinized. Research reinforced that a model's stated reasoning is a generated narrative, not a transparent log of its computation — a convincing chain can sit under a wrong answer.
- Cost awareness grew. Because reasoning tokens are billed and add latency, teams became more deliberate about reserving chain of thought for problems that genuinely need it.
Why step-by-step reasoning helps
For a multi-step problem, the correct answer depends on intermediate results. When a model is forced to produce those intermediates as text, each step conditions the next, so the model is less likely to skip a stage or guess the final answer outright. In effect, generating the reasoning gives the model more computation and a scratchpad to hold partial results.
The catch: this only helps when the task has genuine intermediate structure. For a factual lookup or a simple classification, there is nothing to reason through, and the extra text just adds cost.
When it helps and when it does not
| Task |
Chain of thought |
Why |
| Multi-step math |
Helps |
Intermediate results must be tracked |
| Logic and planning |
Helps |
Sequencing and constraints benefit from steps |
| Code debugging |
Often helps |
Reasoning surfaces the failing path |
| Simple classification |
Usually not |
No intermediate structure to exploit |
| Fact retrieval |
No |
Answer is a lookup, not a derivation |
| Latency-critical calls |
Weigh carefully |
Reasoning adds tokens and time |
Chain of thought and worked examples reinforce each other; a couple of solved examples that show the reasoning steps often outperform a bare instruction. That overlap is covered in few-shot vs zero-shot prompting.
Variants worth knowing
- Zero-shot chain of thought. Simply appending "let us think step by step" triggers reasoning with no examples. Cheap, and still useful on non-reasoning models.
- Few-shot chain of thought. Provide examples that include the reasoning, not just the answer. Strong for teaching a specific solution format.
- Self-consistency. Sample several reasoning paths and take the majority answer. It raises accuracy on hard problems at the cost of multiple generations.
Pitfalls to avoid
- Trusting the narrative. The chain reads like an explanation but is generated text; it can rationalize a wrong answer convincingly. Verify results independently.
- Applying it everywhere. On trivial tasks, forced reasoning wastes tokens and can even introduce errors by overthinking.
- Double-reasoning on reasoning models. If a model already thinks internally, layering a verbose "think step by step" prompt on top is often redundant.
- Ignoring the cost. Reasoning tokens count toward your budget and latency. Measure whether the accuracy gain is worth it for your workload.
FAQ
Does chain of thought make models more accurate?
On problems with real multi-step structure, yes, often substantially. On simple lookups or classification it adds little and can occasionally hurt by encouraging overcomplicated answers.
Is the reasoning the model shows actually how it thinks?
No. It is a plausible narrative generated alongside the answer. It correlates with the model's process but is not a faithful trace, so a sound-looking chain does not guarantee a correct result.
Do I still need it with reasoning models?
Less than before. Models that reason internally already do much of this work, so explicit chain of thought prompting yields smaller gains and is sometimes unnecessary.
What is self-consistency?
It samples multiple independent reasoning paths for the same question and picks the most common final answer. It trades extra compute for improved reliability on difficult problems.
Where to go next