GRPO

Group Relative Policy Optimization (GRPO) was introduced in DeepSeekMath and used in DeepSeek-R1. It keeps the clipped update from PPO but drops the critic. A critic is a value network about the size of the policy, and training it is expensive. GRPO gets the baseline from sibling samples instead.

How it works

For each prompt x, sample a group of G completions from the current policy. Score each one with a reward rᵢ. Then normalize the rewards inside the group:

Aᵢ = (rᵢ − mean(r)) / (std(r) + ε)

Apply a PPO-style clipped policy gradient with these advantages, plus a KL term to a reference model. Completions that beat their group are reinforced. Those that fall short are suppressed.

The group's own mean is the baseline. That is REINFORCE with a prompt-specific baseline, and it comes for free.

Walking through a group

Four solutions to one math prompt score {1, 0, 1, 0}, where 1 means correct.

mean = 0.5
std  = 0.5       (population standard deviation)

A = (1 − 0.5) / 0.5 = +1      (both correct solutions)
A = (0 − 0.5) / 0.5 = −1      (both wrong solutions)

The two correct chains of thought become more likely, and the two wrong ones less likely.

Now take a group where all four are wrong, {0, 0, 0, 0}. The mean is 0 and the standard deviation is 0, so every advantage is 0. No update happens until sampling finds a success.

Why group size matters

Sparse success is the main risk. If only 5% of samples pass, a group of 8 has no winner about two-thirds of the time (0.95⁸ ≈ 0.66). Then the advantages are all zero and learning stalls.

The fixes:

• Use larger groups, so a winner appears more often.
• Raise the temperature early, so samples differ more.
• Choose prompts the base model can sometimes solve, measured by pass at k.
• Bootstrap with supervised examples of worked solutions.

Common mistakes

• Using groups of two or three. The mean and the spread are then too noisy to rank the samples.
• Dropping the KL term. Without it, the policy can drift toward reward-hacking outputs.
• Reading zero advantage as a bug. A group with identical rewards gives no signal, and that is correct behavior.

Check yourself

1. Why does a group where every reward is identical give no learning signal? Answer: after subtracting the group mean, every advantage is 0.
2. What replaces the critic in GRPO? Answer: the average reward of the sibling samples for the same prompt.

Key takeaways

• GRPO uses the mean reward of a group of samples as the baseline, so it needs no critic.
• Advantages are normalized inside each group, so each prompt has its own scale.
• Sparse success stalls learning, so group size, temperature and prompt choice matter.
• It pairs well with verifiable rewards, where a group of answers can be checked cheaply.

Further learning

The DeepSeekMath paper introduces GRPO:

DeepSeekMath (introduces GRPO)Mathematical reasoning poses a significant challenge for language models due to its complex and structured nature. In this paper, we introduce DeepSeekMath 7B, which continues pre-training DeepSeek-Coder-Base-v1.5 7B …arXiv.org

Next, see how the R1 recipe combined this with cold-start data and distillation in the R1 training recipe.