AI Glossary: Chain of Thought

What is Chain of Thought?

Chain of thought (CoT) is a prompting technique introduced by Google Research in 2022 that significantly improves the reasoning capabilities of large language models. The core idea is simple: instead of asking the model to directly output an answer, you instruct it to generate a series of intermediate reasoning steps that lead to the final answer.

Think of it as the difference between a student who just writes down the answer to a math problem and one who shows their work. The student who shows their work is more likely to get the right answer because they're forced to think through each step, and they can catch errors along the way. CoT does the same for AI models.

How Chain of Thought Works

The mechanism behind CoT is both simple and profound. When a model generates tokens, each token is conditioned on all previous tokens. By generating reasoning steps, the model:

1. Decomposes complex problems — Breaking a hard problem into smaller, manageable sub-problems
2. Allocates more computation — More tokens = more forward passes = more "thinking"
3. Provides self-context — Each reasoning step becomes context for the next step, creating a contextual chain
4. Enables error correction — The model can detect and correct inconsistencies in its own reasoning as it continues

Variants: Few-shot, Zero-shot, and Beyond

Few-shot CoT

The original approach: provide a few examples in the prompt, each showing the reasoning chain before the answer. The model learns the pattern of "reason, then answer" from the examples.

Example few-shot CoT prompt (abbreviated):

Q: Roger has 5 tennis balls. He buys 2 more cans. Each can has 3 balls. How many does he have?
A: Roger started with 5 balls. 2 cans × 3 balls = 6 balls. 5 + 6 = 11. Answer: 11 balls.

Q: [Your question here]
A:

Zero-shot CoT

A simpler variant discovered by Kojima et al. (2022): just add "Let's think step by step" to the prompt. No examples needed! This works surprisingly well because the model has seen enough reasoning patterns during pre-training.

Tree of Thoughts (ToT)

Extends CoT by exploring multiple reasoning paths simultaneously. The model generates several possible next steps, evaluates each, and pursues the most promising ones. This allows backtracking and exploring alternatives, much like how humans solve complex problems.

Self-Consistency

Instead of generating one reasoning chain, generate multiple independent chains (using different random seeds or higher temperature), then take a majority vote on the final answer. This is particularly effective for tasks where there are multiple valid reasoning paths.

When to Use Chain of Thought

CoT is not a universal solution. It's most effective for specific types of tasks:

Tasks Where CoT Excels

• Arithmetic reasoning — Multi-step math word problems, calculations requiring multiple operations
• Commonsense reasoning — Questions requiring everyday knowledge and logical inference
• Symbolic reasoning — Tasks involving formal logic, symbol manipulation, or rule application
• Multi-hop QA — Questions requiring combining information from multiple sources or facts
• Code debugging — Tracing through code to find bugs or understand behavior

Tasks Where CoT May Not Help

• Simple factual recall — "What is the capital of France?" doesn't need reasoning
• Creative writing — CoT reasoning may constrain creative output
• Translation — Direct translation usually works fine without explicit reasoning

Practical Examples

Example 1: Arithmetic (Without CoT vs With CoT)

Example 2: Multi-step Logic

With CoT, the model can solve problems requiring multiple inferential steps that would be impossible with direct prompting. For instance, "If all dogs are mammals, and all mammals are animals, and Fido is a dog, what is Fido?" The CoT reasoning: "All dogs are mammals → Fido is a mammal. All mammals are animals → Fido is an animal."

CoT vs Other Reasoning Techniques