Scale and post-training
RLHF and preference optimization
How language models learn from human and AI preferences. Supervised fine-tuning, a reward model, a KL-leashed RL step, and DPO as a single-loss alternative. Start here for the map.
Supervised fine-tuning teaches a model to imitate demonstrations. That is necessary, but it is not enough. Many goals, such as "be concise but complete," "refuse this safely" or "sound like our brand," are easier to express as comparisons than as gold answers. RLHF, reinforcement learning from human feedback, and its relatives turn preference data into a training signal. DPO folds much of that pipeline into a single loss.
This section sits on RL basics: policies, rewards, KL and PPO. Productized safety continues in guardrails.
The InstructGPT pipeline
OpenAI's InstructGPT paper set out a three-stage loop that the industry still uses, with variations.
Stage 1 trains the supervised model. It becomes both the starting point for RL and the reference the KL penalty pulls toward. Stage 2 trains a reward model on pairs of replies. Stage 3 optimizes the policy against that reward model, with the KL leash in place.
Concepts in this section
Read them in order. Each step uses the one before it.
| Step | Concept | What it answers |
|---|---|---|
| 1 | Preference pairs and Bradley-Terry | Why compare replies, and how does a reward model learn from pairs? |
| 2 | The KL leash in RLHF | How does the policy improve against a reward without gaming it? |
| 3 | DPO | Can preference training skip the reward model and the RL loop? |
| 4 | RLAIF and Constitutional AI | How do models provide feedback at a scale humans cannot reach? |
Common mistakes
Preference optimization moves the model toward the label distribution. It does not reach an objective truth. These failures show up often.
| Failure | What you see | Mitigation |
|---|---|---|
| Reward hacking | High reward-model score, worse human eval | Stronger KL, better reward data, periodic human eval |
| Sycophancy | Model agrees with the user's false claims | Preference data that rewards truth over agreement |
| Over-optimization | The proxy rises, true quality falls | Stop on a gold win rate, mix in supervised data |
| Verbosity bias | Longer answers win the labels | Length-aware labels or a length penalty |
| Mode collapse | The same canned openings everywhere | Entropy, higher KL, more diverse prompts |
Warning
As the reward-model score climbs during training, human win rate can peak and then fall. Plot both. Blindly maximizing the proxy is how you ship a model that agrees with everything the user says.
Maintain a frozen gold set of preferences that the training never uses. Report the win rate against the supervised model on that set. If the training reward rises while the gold win rate falls, stop and fix the data or the leash.
Key takeaways
- Preferences scale better than demonstrations for many alignment goals. Supervised fine-tuning comes first, then preference optimization.
- InstructGPT runs supervised fine-tuning, then a Bradley-Terry reward model, then PPO with a KL penalty.
- DPO optimizes the same KL-constrained goal with a closed-form loss, with no separate reward model.
- AI feedback scales labeling, but it inherits its labeler's biases and needs human checks.
- Watch reward hacking, sycophancy and over-optimization. Track human win rate, not only the reward score.
Further learning
Shaw Talebi connects InstructGPT, PPO and DPO, then fine-tunes a small model on real preference pairs:
Direct Preference OptimizationWhile large-scale unsupervised language models (LMs) learn broad world knowledge and some reasoning skills, achieving precise control of their behavior is difficult due to the completely unsupervised nature of their t…arXiv.org
RLHF Book (Nathan Lambert)A free online book and course on RLHF, preference tuning, reward models, RLVR, and post-training language models.RLHF Book
Related
Reinforcement learning basics
MDPs, returns, policy gradients, advantages and PPO. The minimum RL needed to read modern LLM post-training papers. Start here for the map.
Supervised fine-tuning and PEFT
Instruction tuning, chat formats, data quality, LoRA/QLoRA math, and the tooling that turns a base model into a useful assistant.
Guardrails, security and safety
Prompt injection is unsolved. Cut blast radius with architecture, not a classifier that claims to detect it.