Chain of thought and test-time compute

Some problems are too hard to answer in one short step. A model that writes out its working before the final answer often does better. That written working is chain of thought (CoT). The more tokens a model is allowed to spend on a query, the more room it has to work things out. Spending that budget well is the idea behind reasoning models.

Why extra tokens can help

Each intermediate token is a choice the model makes. A reasoning trace can plan, check a sub-result, backtrack and try again. For a hard math problem, the final answer is only reachable through those steps.

Two curves describe the trade-off:

• Train-time scaling asks how accuracy changes as you train on more data and parameters. See scaling laws.
• Test-time scaling asks how accuracy changes when the weights stay fixed and the model thinks longer or samples more.

Simple strategies before any RL

You can get gains without changing the weights:

• Self-consistency. Sample several reasoning chains and take a majority vote on the final answer.
• Budget forcing. Ask the model to keep thinking, up to a token limit.
• Tool loops. Let the model call a calculator, a search or a code runner inside its reasoning.

Here is self-consistency on five samples of one problem, with final answers 42, 42, 37, 42, 40. The answer 42 appears three times, so it wins the vote. The vote only works when answers can be compared, such as numbers, short strings or code outputs.

RL goes further. It changes the weights, so the first sample is more often a good long reasoner, rather than relying on a vote over many weak ones.

Common mistakes

• Assuming more tokens always help. Past a point, extra thinking adds cost and can add errors.
• Using voting with free-form answers. Votes only count when the final answers match. Normalize or extract the answer first.
• Reading a long, reflective trace as proof of accuracy. A model can think out loud and still be confidently wrong.

Check yourself

1. Why does self-consistency need comparable final answers? Answer: votes only add up when answers match exactly, so the answers must be extracted and normalized.
2. What does RL add beyond voting over samples? Answer: it changes the weights, so the model's first attempt is more often a good one.

Key takeaways

• Chain of thought is intermediate reasoning that the model writes before its answer.
• Test-time compute is the extra inference work spent per query, and it can improve hard problems.
• Self-consistency and budget forcing help before any training.
• RL trains the model to use its budget well.

Further learning

Next, see which rewards can judge the answers a reasoning model writes, in verifiable rewards.