Verifiable rewards

Preference models judge tone and helpfulness, but they can be fooled. For math and code, you often do not need a judge at all. You can check the answer. A verifiable reward is a rule that returns 1 for a correct answer and 0 otherwise, with no opinion involved. Reasoning RL leans on these rewards wherever it can.

Verifiable or learned

Use verifiers whenever ground truth exists. Fall back to learned rewards for open-ended quality, and keep a KL leash on the policy so it cannot drift far.

A typical composite reward

Math training usually adds a few terms:

r = r_correct + λ_fmt · r_format − λ_len · length_penalty

• r_correct is 1 when the extracted answer matches, and 0 otherwise. Normalize both sides first, so 42 and 42. count as the same answer.
• r_format rewards a parseable layout, such as a final line the grader can read.
• length_penalty discourages padding.

Walk through the length pressure with made-up weights, λ_fmt = 0.1 and λ_len of 0.001 per token. A correct, well-formatted answer in 500 tokens scores 1 + 0.1 − 0.5 = 0.6. The same answer in 200 tokens scores 1 + 0.1 − 0.2 = 0.9. The penalty is small enough to keep the reasoning, and large enough to stop waffle.

For code, run hidden unit tests in a sandbox with CPU and memory limits. Reward the pass rate. Flaky tests add noise to the reward, which can look like an algorithm failure.

Common mistakes

• Using a loose parser. The model learns the parser's quirks, such as a format that passes without correct reasoning.
• Ignoring flaky tests. Non-deterministic graders make the reward noisy.
• Rewarding length without a cap. Reward curves can rise while answers get longer and worse. Watch mean length alongside accuracy.

Check yourself

1. Why normalize answers before the exact-match check? Answer: so trivial formatting differences do not count as wrong answers.
2. When would you use a learned reward? Answer: when no checker exists, such as open-ended writing quality, and then with a KL leash.

Key takeaways

• A verifiable reward checks the answer. It is the cleanest reward a language model can get.
• Reward design combines correctness, format and length, and each term needs care.
• Parsers, flaky tests and unchecked length are the common failure points.
• Learned rewards are a fallback, and they need a KL leash.

Further learning

Next, see how these rewards feed a group-based method in GRPO.