PPO and the KL leash

Policy gradients can take a large step after one batch of samples, and a large step can wreck the policy. Proximal Policy Optimization (PPO) is the actor-critic method that became the workhorse of classic RLHF. Its key idea is to avoid trusting a big update from one batch.

The probability ratio

PPO compares the new policy with the one that produced the samples:

ρ(θ) = πθ(a | s) / πθold(a | s)

A ratio of 1 means no change. A ratio of 1.5 means the action is now 50% more likely than before.

The clipped objective

PPO maximizes this:

L = E[ min( ρ · Â, clip(ρ, 1 − ε, 1 + ε) · Â ) ]

Here Â is the advantage, and ε is usually about 0.2. Walk through one case, with  > 0 and ρ = 1.5:

unclipped term = 1.5 · Â
clipped term   = clip(1.5, 0.8, 1.2) · Â = 1.2 · Â
objective      = min(1.5 · Â, 1.2 · Â) = 1.2 · Â

Once the ratio passes 1.2, the objective stops rewarding the extra change. The policy gains nothing by moving further in one update. When the advantage is negative, the min keeps the more pessimistic term, so the update still cannot earn credit for overshooting.

That limit is the "proximal" part. It keeps each update close to the old policy.

The KL leash

In LLM training, the policy is also pulled toward a reference model, usually the supervised checkpoint. The penalty is the KL divergence between the two, measured per token:

• Without it, a strong optimizer and an imperfect reward move probability to odd high-scoring habits, such as empty answers or repeated keywords.
• With it, the policy keeps the fluency and knowledge of the reference model.

Teams track the mean KL per token during training. If it spikes, they back off the learning rate or raise the penalty. The RLHF page covers the penalty in more depth.

PPO or a simpler method

PPO needs a critic, which is about as large as the policy, and online sampling. That is expensive but well tested.

Group-based methods drop the critic and use the group's own average as the baseline. They can be enough when rewards come at the end of a completion and you can sample several completions per prompt. GRPO is the best-known example.

Start simple, and add a critic only when the variance is still too high.

Common mistakes

• Thinking the clip is the whole safeguard. It limits one update. Watch the KL across the run.
• Setting ε without checking. A clip range that is too wide lets updates overshoot. Tune it against the KL curve.
• Watching only reward. Reward can rise while the outputs get worse. Plot KL, entropy and sample length with it.

Check yourself

1. With ε = 0.2, what range does the ratio get clipped to? Answer: [0.8, 1.2].
2. Why does a critic help? Answer: it estimates the value of each state, so advantages do not need to wait for the full return.

Key takeaways

• PPO limits each update by clipping the probability ratio to [1 − ε, 1 + ε].
• Clipping stops reward gains from one update, but over many updates the policy can still drift.
• A KL penalty to a reference model keeps the policy close to its starting point.
• PPO needs a critic. Group-based methods like GRPO can skip it.

Further learning

Next, see how these ideas map onto language models in RL for LLMs.