Scale and post-training
Process vs outcome rewards
An outcome reward scores only the final answer. A process reward scores each step. Outcomes are simpler and the default, while process rewards give finer credit at a higher labeling cost.
A long reasoning trace has many steps, and only some of them matter. The question is where the reward goes. An outcome reward model (ORM) scores only the final answer. A process reward model (PRM) scores each intermediate step, asking whether this line of reasoning is valid. The choice decides how precisely the model learns which steps were good.
The credit problem
Suppose a 4-step solution reaches the right final answer, but step 2 contains an error. An outcome reward gives every step the same credit, including the bad one. The model is then rewarded for a flawed step, because the answer happened to be right.
A process reward can score the steps separately. For instance, it might output 0.9, 0.2, 0.8, 0.9 for the four steps. The model can then learn that step 2 was the problem.
Over a 10,000-token chain of thought, outcome rewards make credit assignment very hard. Process rewards make it finer, but they need labels for each step.
Walking through the trade-off
| Outcome reward | Process reward | |
|---|---|---|
| What it scores | Final answer or final state | Each intermediate step |
| Signal | Simple and cheap to produce | Richer, but step labels are costly |
| Main risk | Rewards lucky answers with bad steps | A weak step scorer becomes a target to game |
| Typical use | Math and code RL with checkers | Search, and domains where steps can be graded |
Trade-off
Outcome verifiers scale with automatic checkers, so they are the default for math and code RL. Process models help most in search, where a tree or beam search is guided by step scores, and in domains where final answers are rare but steps can be graded.
Many production systems still succeed with outcome verifiers alone. They rely on enough samples for GRPO to find the winners.
A hybrid is common: train the policy with outcome rewards, and use a light PRM only at inference to prune bad partial thoughts.
Common mistakes
- Training a PRM without reliable step labels. A bad step scorer is another surface the policy can exploit.
- Assuming process rewards are free rigor. They can reward long, confident-looking steps that add nothing.
- Starting with a PRM by default. Begin with an outcome verifier. Add step scores only where outcomes are too sparse.
Check yourself
- Why can an outcome reward reinforce a wrong step? Answer: every step shares the final reward, so a lucky correct answer rewards its flawed steps too.
- Why is a PRM a risk? Answer: the policy will find whatever the step scorer rewards, including steps that look rigorous but are not.
Key takeaways
- Outcome rewards score the final answer. Process rewards score each step.
- Outcome rewards are simple and are the default for math and code.
- Process rewards give finer credit, at the cost of step labels and a new target to game.
- A common hybrid trains with outcomes and uses a light step scorer to prune at inference.
Further learning
Julia Turc builds GRPO from REINFORCE and PPO intuition, without assuming an RL background:
Next, see the overview of the whole section in RL for reasoning models.
Related
Verifiable rewards
Reasoning RL works best when a checker can say whether an answer is right. Exact answers, unit tests and proof checkers give the cleanest reward a language model can get.
GRPO
Group Relative Policy Optimization samples several completions per prompt and uses their average reward as the baseline. It drops the value network that PPO needs.
RL for reasoning models
How reasoning models learn to think before they answer, using verifiable rewards, GRPO and test-time compute. Start here for the map.