In 2022, two short ideas changed how people use language models for reasoning. Chain-of-thought prompting showed that asking a model to write out intermediate steps dramatically improved its performance on maths and logic. Self-consistency showed that sampling several such reasoning chains and taking the most common final answer improved results further. Together they laid the groundwork for today's reasoning models, which generate long internal chains of thought by default.
This article revisits both techniques: what the original papers found, why they work, their limits (including whether written reasoning reflects what the model actually does), and how to use them now that reasoning models exist.
Chain-of-thought prompting
Wei et al. (2022), "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models," gave models few-shot examples that included worked reasoning before each answer. On arithmetic word problems such as the GSM8K benchmark, symbolic reasoning and commonsense tasks, this produced large improvements for sufficiently large models; smaller models benefited little or even got worse.
Kojima et al. (2022), "Large Language Models are Zero-Shot Reasoners," showed that you did not even need examples: appending "Let's think step by step" to the question produced a similar, though smaller, effect.
Why it works
Several explanations complement each other:
- More computation per answer. Each generated token is another forward pass through the model. Writing steps gives the model more computation before committing to an answer.
- Intermediate results become inputs. Once "17 × 4 = 68" is written, later steps can use 68 directly instead of computing it implicitly.
- Decomposition. Writing steps encourages breaking the problem into parts, similar to decomposition prompts for long tasks.
The gains concentrate on tasks with genuine intermediate steps. For many short, everyday tasks, the benefit is small; see does "think step by step" help on short tasks?.
Self-consistency
Wang et al. (2022), "Self-Consistency Improves Chain of Thought Reasoning in Language Models," made a simple observation: a hard problem can be reached by different reasoning paths, and correct paths tend to converge on the same answer, while errors scatter. So instead of taking one greedy chain:
- Sample many reasoning chains at a moderate temperature.
- Extract the final answer from each.
- Take the majority vote.
On arithmetic and commonsense benchmarks, self-consistency improved substantially over single-chain chain-of-thought. It requires no training, works with any model and is easy to implement.
from collections import Counter
def self_consistent_answer(llm, question, n=8, temperature=0.7):
answers = []
for _ in range(n):
out = llm(f"{question}\nThink step by step, then give the final answer after 'Answer:'.",
temperature=temperature)
if "Answer:" in out:
answers.append(out.split("Answer:")[-1].strip().lower())
if not answers:
return None, 0.0
best, count = Counter(answers).most_common(1)[0]
return best, count / len(answers) # answer and agreement rateThe agreement rate is a useful by-product: when samples disagree a lot, the question is hard or ambiguous, which is a good signal for escalation to a stronger model or a human. Temperature matters here; see LLM temperature explained.
Limits of self-consistency
- Cost scales linearly with the number of samples.
- It needs comparable answers. Majority voting works for numbers, labels and short answers. For free-form text, variants such as universal self-consistency ask a model to pick the most consistent response among samples.
- Shared errors survive. If the model has a systematic misconception, most samples will share it, and the vote will confirm it.
Beyond a single chain
Later work explored richer search over reasoning: Tree of Thoughts (Yao et al., 2023) let models branch, evaluate partial solutions and backtrack, which helped on puzzle-like tasks with clear intermediate evaluation. Self-critique methods ask the model to review its chain, with mixed results; see the red-team yourself prompt.
Is the written reasoning faithful?
An important caveat: a chain of thought is text the model generates, not a window into its computation. Research has shown that stated reasoning does not always reflect what drives the answer. Turpin et al. (2023), "Language Models Don't Always Say What They Think," found that models could be biased by features of the prompt, such as the order of answers in few-shot examples, while producing plausible explanations that never mentioned the bias. Lanham et al. (2023) studied how much models actually rely on their stated reasoning, with results varying across tasks and model sizes.
Practical implications:
- treat reasoning as useful evidence, not a guaranteed explanation;
- verify important conclusions independently, especially in high-stakes settings;
- do not assume a model "did not consider" something because its reasoning omits it.
How reasoning models changed the picture
Since 2024, many models are trained, often with reinforcement learning on problems with verifiable answers, to produce extended reasoning before answering. In effect, chain of thought became a trained behaviour instead of a prompting trick, and models learned to explore, check and revise within their reasoning.
What this means in practice:
| Technique | Standard model | Reasoning model |
|---|---|---|
| "Think step by step" | Often helps on multi-step tasks | Usually unnecessary; use effort settings |
| Few-shot reasoning examples | Can help format and approach | Often less useful; may constrain reasoning |
| Self-consistency (majority vote) | Strong gains on hard problems | Still helps on the hardest problems, at high cost |
| Agreement rate as confidence | Useful signal | Useful signal |
Reasoning models also make cost a bigger concern, because reasoning tokens are billed; see what reasoning tokens actually cost you. Whether to buy reasoning with a reasoning model or with self-consistency over a cheaper model is an empirical question for each task.
Getting the answer out reliably
Both techniques depend on extracting a final answer from free text, and extraction errors are a common hidden source of bad results. Ask for the answer after a fixed marker such as "Answer:", normalise answers before voting (case, whitespace, units, number formats), and treat outputs without the marker as failures rather than guessing. For structured tasks, have the model reason in free text and then emit the answer in a small structured block; the patterns in structured output without JSON mode apply directly. When voting, log every sample's answer, not only the winner, so you can see how close each vote was.
Practical recommendations
- Use chain of thought for multi-step problems with standard models; skip it for simple recognition tasks.
- Use self-consistency for high-value, verifiable answers, especially when you can afford several samples, and use agreement to route uncertain cases.
- With reasoning models, control reasoning through effort settings rather than prompt instructions.
- Verify outcomes with tools or checks where possible: calculators for arithmetic, tests for code, sources for facts.
- Measure gains and costs on your own eval set before adopting any technique broadly.
Frequently asked questions
What is chain-of-thought prompting?
Chain-of-thought prompting asks a language model to write intermediate reasoning steps before giving its final answer, either through worked examples in the prompt or an instruction such as "Let's think step by step." It improves performance mainly on multi-step reasoning tasks.
What is self-consistency in LLMs?
Self-consistency samples several independent reasoning chains for the same question, extracts each final answer and selects the most common one. It often improves accuracy on reasoning tasks compared with a single chain.
Is a model's chain of thought an accurate explanation of its reasoning?
Not necessarily. Research has shown that stated reasoning can omit factors that actually influenced the answer. Treat it as useful evidence and verify important conclusions independently.
Do reasoning models still need chain-of-thought prompts?
Usually not. Reasoning models generate their own extended reasoning; adding instructions to think step by step tends to add length without improving accuracy. Use their reasoning-effort settings instead.
