Reinforcement learning basics

Supervised learning needs labeled answers. Reinforcement learning (RL) learns from rewards: scalar feedback that may arrive only at the end of a long sequence of decisions. That is the shape of LLM post-training. A whole completion is scored as more or less preferred, correct or safe, and the vocabulary of policies, returns and advantages shows up in every RLHF and reasoning paper.

You do not need a whole textbook to read a PPO or GRPO objective without freezing. You need the core ideas, and this section gives them with language models in mind. Deeper classical RL, such as Atari and robotics, is left to the further reading at the end.

In one sentence: the policy is what you train, the reward is what you measure, and the advantage tells you which samples to reinforce.

Why RL for language models

Preference data and verifiers do not give a correct answer to copy. They give a score for a whole reply. A method that can learn from scores, and assign credit across many tokens, is the only way to optimize that score directly. The concepts below build that method one piece at a time.

Concepts in this section

Read them in order. Each one builds on the one before.

Common mistakes

• Reading reward as ground truth. A reward is a proxy. The policy will find the gaps in it. This is reward hacking, the central failure of RL. Examples include verbose waffle that fools a preference model, or sycophantic agreement. Stronger optimizers make hacking more likely, so inspect samples as reward rises.
• Watching reward alone. Plot reward, KL to the reference, entropy and sample length together. Reward alone can hide a collapsing policy.
• Skipping the baseline. Without one, the gradient is noisy and learning is slow.

Key takeaways

• RL learns from rewards in a loop of state, action, next state and reward.
• A return is the discounted sum of future rewards. Advantages compare an action with the average.
• Policy gradients raise the probability of actions in proportion to their advantage. Baselines cut the noise.
• PPO clips the probability ratio, and a KL penalty keeps the policy close to a reference model.
• For LLMs, the policy is the model, tokens are actions and a completion is the episode.

Further learning

David Silver's first lecture is the classic conceptual intro, covering agent, reward and value:

Spinning Up in Deep RL (OpenAI)Welcome to Spinning Up in Deep RL! — Spinning Up documentationspinningup.openai.com Reinforcement Learning: An Introduction (Sutton & Barto)Sutton & Barto Book: Reinforcement Learning: An Introductionincompleteideas.net Proximal Policy Optimization AlgorithmsWe propose a new family of policy gradient methods for reinforcement learning, which alternate between sampling data through interaction with the environment, and optimizing a "surrogate" objective function using stoc…arXiv.org