RL for language models

Once you know policies, returns and advantages, the next step is applying them to a language model. The mapping is direct. The model is the policy. Each next token is an action. The prompt and the completion together form one episode. The hard parts are the same ones RL always has: giving credit to the right choices and keeping the model exploring.

The mapping

Most algorithms treat the whole completion as one unit that gets a single reward at the end. That simplifies training, but the signal is coarse.

Credit assignment

When the reward comes only at the end, every token in a completion shares the same advantage in the simplest method. An early token that set up a good plan gets the same weight as a lucky final word.

Here is why that matters. Three completions score {0, 1, 0}. With a baseline of 1/3, the advantages are −1/3, +2/3, −1/3. The good completion is pushed up, and the bad ones are pushed down. Without a baseline, the two losers would contribute nothing.

Methods such as process rewards, value functions and group-relative baselines exist to sharpen this credit. Reasoning training with sparse rewards needs many samples per prompt, so that some completions succeed and can be reinforced.

Exploration

Exploitation picks the best-looking action so far. Exploration tries something else to learn more.

In language model RL, sampling temperature and an entropy bonus play the role of exploration. Too little exploration and the policy collapses onto a few safe templates. Too much and the reward never climbs.

A practical schedule starts with higher temperature while the policy finds good modes. Then it cools down, so serving can use a lower temperature without a mismatch between training and serving.

Watch for silent collapse. Mean reward can look fine while the outputs become repetitive. Log the diversity of rollouts, such as the unique n-gram rate, along with the reward.

Off-policy and on-policy

On-policy methods, like classic PPO, need samples from the current policy. Once the policy changes, old samples no longer fit.

Off-policy methods reuse data from older policies, using correction terms. DPO is often described as offline preference learning. It trains on a fixed set of pairs without sampling in the inner loop.

Real pipelines mix both. Offline preference steps are cheap for alignment. On-policy RL is used when the policy must explore against a verifier.

Common mistakes

• Treating the completion as many independent choices. The reward applies to the whole reply, so credit is shared.
• Watching only the reward curve. Reward can rise while diversity falls. Plot several signals together.
• Expecting exploration to take care of itself. Temperature and entropy need an explicit schedule.

Check yourself

1. Why does a baseline help when three completions score 0, 1, 0? Answer: it makes the losers' advantages negative, so they are pushed down too, not left at zero.
2. Why cool the temperature after training? Answer: so serving samples at a lower temperature without the training and serving distributions drifting apart.

Key takeaways

• The model is the policy, tokens are actions, and a completion is the episode.
• Credit assignment is the hard part. A whole completion usually gets one reward.
• Exploration comes from temperature and entropy. Schedule it, and watch diversity.
• Offline preference methods and on-policy RL are complementary.

Further learning

Next, see how preference data trains a reward signal in RLHF and preference optimization. For reasoning models that use verifiable rewards, see RL for reasoning models.