"Let's think step by step" is probably the most famous phrase in prompt engineering. Adding it, or asking a model to explain its reasoning before answering, became standard advice after research showed large gains on maths and logic problems. Many teams now include some version of it in every prompt.
But most production tasks are not maths problems. They are short: classify this message, extract this date, rewrite this sentence, answer this question from the provided text. For tasks like these, does asking for reasoning help, or does it just make responses slower and more expensive? This guide summarises what the research shows, explains why the effect varies so much by task, and gives you a simple way to test it on your own workload.
What chain-of-thought prompting is
Chain-of-thought (CoT) prompting asks the model to write out intermediate reasoning steps before the final answer. There are two common forms:
- Few-shot CoT (Wei et al., 2022): include examples in the prompt that show worked reasoning before each answer.
- Zero-shot CoT (Kojima et al., 2022): simply add an instruction such as "Let's think step by step."
Both papers reported large improvements on arithmetic, symbolic and multi-step reasoning benchmarks, especially for larger models. That is where the phrase's reputation comes from.
What the research says about other tasks
Later work looked more broadly. Sprague et al. (2024), "To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning," analysed results across many papers and ran their own experiments across a wide range of tasks. They found that CoT produced strong gains mainly on maths and symbolic reasoning tasks, and much smaller or negligible gains on many other task types, such as commonsense and knowledge questions. They also observed that much of the benefit on maths came from tasks that involve explicit calculation or symbol manipulation, steps that the written reasoning makes easier to carry out.
Two more developments changed the picture:
- Reasoning models now perform extended reasoning internally by default. Asking them to "think step by step" often adds little, and can increase output length without improving the answer.
- Stronger base models solve many simple tasks correctly without visible reasoning, leaving less room for CoT to help.
Why reasoning helps some tasks and not others
Writing out steps helps when the task requires intermediate results that the model cannot easily compute in a single step: multi-step arithmetic, tracking several constraints, combining facts from different places, or following a procedure.
For short tasks where the answer depends on recognising a pattern, such as sentiment, topic, a field present in the text or a rewrite in a given style, there are no intermediate results to compute. The reasoning text becomes a description of the answer rather than a path to it.
| Task type | Expected benefit | Why |
|---|---|---|
| Multi-step maths, calculations | High | Intermediate results must be computed |
| Logic puzzles, constraint tracking | High | Steps keep constraints visible |
| Multi-hop questions over documents | Moderate | Combines facts from several places |
| Classification with clear categories | Low | Recognition, not computation |
| Field extraction | Low | Answer is present in the input |
| Rewriting, translation, tone changes | Low to none | Pattern-heavy generation |
| Ambiguous classification with rules | Moderate | Applying rules can benefit from steps |
The costs of asking for reasoning
Even when accuracy does not change, reasoning has costs:
- Tokens. Reasoning can be several times longer than the answer, multiplying output cost. For high-volume tasks this adds up quickly; see what reasoning tokens actually cost you.
- Latency. Users wait for reasoning before the answer appears.
- Parsing. You need to separate reasoning from the answer reliably.
- Occasional harm. On some tasks, reasoning can talk the model out of a correct intuitive answer or introduce errors, the same right-to-wrong flips seen with self-critique prompts.
There are also benefits beyond accuracy. Visible reasoning helps with debugging and auditing, and can make it easier to spot why a model went wrong. If that matters for your use case, you may keep reasoning even without an accuracy gain, but decide deliberately.
How to test it on your own task
- Pick a representative set of 100 to 200 inputs with known correct outputs.
- Create two prompt variants that are identical except for the reasoning instruction. In the reasoning variant, require the final answer in a clearly marked section.
- Run both at your production temperature, several times per item if temperature is above 0.
- Measure accuracy, output tokens and latency for each variant.
- Compute a confidence interval for the accuracy difference before concluding; small differences on small sets are often noise; see why your 2-point improvement might be noise.
- Read the flips: items correct only with reasoning, and items correct only without. The pattern tells you which sub-types of your task benefit.
A short, bounded reasoning instruction like this, limited to two sentences and focused on the deciding evidence, is often a good compromise: it keeps much of the debugging value at a fraction of the token cost.
Hidden reasoning vs. visible reasoning
With modern APIs you often have a choice of where reasoning happens. You can ask a standard model to write its reasoning in the response, or you can use a reasoning model whose internal reasoning is hidden or summarised and controlled by an effort setting.
The two behave differently in practice:
- Visible reasoning is fully under your control: you can bound its length, specify what it should cover and log it for audits. It also appears in the response, so you must strip it before showing the answer to users.
- Internal reasoning is usually stronger on hard problems, because the model was trained to use it, but you see less of it and pay for all of it. Lowering the effort setting is the main lever for cost.
For short, simple tasks, neither is usually needed. For hard tasks, prefer a reasoning model with an appropriate effort setting over long hand-written reasoning instructions to a standard model. For audit trails, a short, bounded visible justification is often enough.
Practical recommendations
- Use CoT by default for maths, multi-step logic, multi-hop questions and rule-heavy decisions.
- Skip it by default for simple classification, extraction and rewriting, and test before adding it.
- Don't stack it on reasoning models: they already reason; extra instructions mostly add length.
- Bound it when you keep it for auditability: limit length and ask for the decisive evidence only.
- Consider sampling instead: on hard reasoning problems, several samples with a majority vote can outperform a single long chain; see chain of thought and self-consistency.
Frequently asked questions
Does "let's think step by step" still work?
It still helps on tasks that need multi-step reasoning, such as maths and logic, especially with non-reasoning models. On simple tasks and with reasoning models, the benefit is usually small or absent.
When should I use chain-of-thought prompting?
Use it when a task requires intermediate steps: calculations, tracking multiple constraints, or combining facts from different parts of a document. Test before using it on classification, extraction or rewriting.
Does chain-of-thought increase cost?
Yes. Reasoning text is billed as output tokens and can be many times longer than the answer, and it adds latency. Measure whether the accuracy gain justifies the cost for each task.
Should I ask a reasoning model to think step by step?
Usually not. Reasoning models already generate internal reasoning. Additional instructions tend to lengthen output without improving accuracy; use the model's reasoning-effort settings instead.
