Chain-of-thought and extended thinking

Chain-of-thought (CoT) means the model writes out intermediate reasoning before its final answer. On multi-step problems, that scratch work often lifts accuracy. The original paper showed large gains on arithmetic and multi-step reasoning when models were prompted to think step by step.

The big idea

Reasoning helps when the task has several steps, hidden constraints or a decision between alternatives. It does not help when the task is a pure format change, such as turning a list into JSON. There, the extra tokens only add cost and latency.

Many frontier models already reason internally when the task needs it. Some APIs expose this as a thinking setting or a budget. Use the native feature when you have it. Prompt for visible reasoning only when you have no native option.

Walking through it

Take a refund rule: a damaged item is refundable if the order shipped less than 14 days ago. An order shipped 6 days ago, and the item is damaged.

Step 1: days since shipping = 6
Step 2: window = 14 days
Step 3: days left in the window = 14 - 6 = 8
Step 4: reason = damaged, which the policy allows
Answer: eligible, within the window

Each step is one small check. A model that jumps straight to "eligible" may skip a step. Asking for the steps makes the check visible, and makes it easier to find where a wrong answer went off track.

When you must prompt for reasoning yourself, separate the scratchpad from the answer with tags:

Work through the trade-offs inside <thinking>...</thinking>.
Put only the final recommendation inside <answer>...</answer>.

Your code reads only the <answer> block. The reasoning stays available for debugging.

Prompt chaining

Sometimes one prompt is too much. Split the job into stages, each with its own call:

1. Extract candidate facts from the document.
2. Check and normalize those facts in code.
3. Write the user-facing answer from the checked facts only.

Chaining costs more latency and tokens. It pays off when each stage can be tested and a bad fact from step 1 cannot leak into the final prose. Use it when your evals show mixed-up failures.

Common mistakes

• Reasoning on format tasks. Asking for steps to convert a list to JSON adds cost and gains nothing.
• Treating visible reasoning as proof. The written steps are not a faithful record of the model's internal computation. Use them to debug, and check the answer separately.
• Parsing the answer out of mixed text. Without tags, your code cannot tell reasoning from the answer.

Check yourself

• Why separate <thinking> from <answer>? So your code reads only the final answer and the reasoning cannot leak into the output.
• Should you add "think step by step" to a model that already has native thinking? Usually no. Use the provider's thinking setting instead.

Key takeaways

• Chain-of-thought helps on multi-step decisions and hurts on pure format tasks.
• Prefer native thinking settings when the API offers them.
• Keep scratch work and the final answer in separate, tagged parts.
• Split hard jobs into chained calls when the stages need separate tests.

Further learning

The paper that established chain-of-thought as a technique is Chain-of-Thought Prompting Elicits Reasoning in LLMs.

Next, go back to the prompting hub for the cross-cutting mistakes and the eval loop.