Scale and post-training
RL for reasoning models
How reasoning models learn to think before they answer, using verifiable rewards, GRPO and test-time compute. Start here for the map.
Preference RLHF makes a model more helpful and polite. It does not, by itself, turn a middling math model into one that solves a hard contest problem by reasoning for a few minutes and then gives the right answer. Reasoning models are trained to spend extra compute on chain of thought, and they are reinforced with rewards you can check: unit tests, exact numeric answers and compilers. Algorithms such as GRPO make that affordable. DeepSeek-R1 made the recipe public, and its ideas now shape the frontier (LLM landscape).
The core claim is simple. If a reward can be checked, the training signal is clean, and the model can learn to reason longer and better.
Concepts in this section
Read them in order. The first two cover the setup, and the rest build on it.
| Step | Concept | What it answers |
|---|---|---|
| 1 | Chain of thought and test-time compute | Why does writing out reasoning help, and how do you spend more compute per query? |
| 2 | Verifiable rewards | Which rewards can you check, and how do you combine them? |
| 3 | GRPO | How do you get a baseline without a critic? |
| 4 | The R1 training recipe | How do RL, supervised data and distillation fit together? |
| 5 | Process vs outcome rewards | Should the reward score the answer or each step? |
Common mistakes
- Trusting reward curves alone. If the mean length explodes while accuracy stays flat, the reward is paying for verbosity. Add a length penalty or a format constraint.
- Confusing reflection with correctness. A model that thinks out loud can still be confidently wrong. Evaluate final answers under a token budget.
- Relying on RL alone in regulated domains. Keep a human or a symbolic checker in the loop.
Open problems
- Verifier hacking. Models can produce answers that pass the checker without sound reasoning, such as guessing the expected format or exploiting test cases. Hidden tests help, but they do not end the arms race.
- Overthinking. Models can learn that longer answers score higher, even when a short solution is enough. Cost and latency then grow.
- Transfer. RL on math and code does not automatically fix soft reasoning in law or medicine. Those domains need their own verifiers.
- Agentic RL. When tools such as search or code execution are part of the episode, the setup is multi-step. Cap episode length, and reset the environment between rollouts.
Key takeaways
- Reasoning models learn chain-of-thought policies, reinforced by rewards that can be checked.
- Verifiable rewards beat hackable reward models whenever ground truth exists.
- GRPO uses the group's mean reward as its baseline, which removes the critic.
- The R1 recipe mixes pure RL, cold-start data for readability, more RL, and distillation for cheaper serving.
- Watch for verifier hacking, overthinking and the cost of long traces.
Further learning
Julia Turc builds GRPO from REINFORCE and PPO intuition:
Yacine's overview maps the full R1 pipeline, from Zero to cold start to distillation:
DeepSeekMath (introduces GRPO)Mathematical reasoning poses a significant challenge for language models due to its complex and structured nature. In this paper, we introduce DeepSeekMath 7B, which continues pre-training DeepSeek-Coder-Base-v1.5 7B …arXiv.org
RLHF Book (Nathan Lambert)A free online book and course on RLHF, preference tuning, reward models, RLVR, and post-training language models.RLHF Book
Related
RLHF and preference optimization
How language models learn from human and AI preferences. Supervised fine-tuning, a reward model, a KL-leashed RL step, and DPO as a single-loss alternative. Start here for the map.
Scaling laws
How loss scales with parameters, data and compute, and what Chinchilla and test-time compute changed.
The LLM landscape and model choice
Frontier vs open-weight models, reasoning models, and a process for picking one with your own evals.