Direct preference optimization (DPO)

The full RLHF pipeline trains a reward model, then runs PPO against it. Direct Preference Optimization (DPO) gets to a similar place in one step. It starts from the same KL-constrained goal, and it shows that the best policy can be written in terms of the policy itself. The result is a loss you can train with ordinary supervised machinery on preference pairs.

The loss

For a pair with preferred reply y_w and rejected reply y_l, DPO minimizes:

L_DPO = −log σ( β · [ log π_θ(y_w | x) / π_ref(y_w | x)
                    − log π_θ(y_l | x) / π_ref(y_l | x) ] )

In plain words: raise the likelihood of the preferred reply and lower the likelihood of the rejected one. Both are measured relative to the reference model. The factor β acts as the KL strength, as in the RLHF objective.

Walking through the intuition

Take one pair. Before training, the policy and the reference give the same likelihood to both replies. The bracket is zero, so the loss is −log σ(0) = log 2 ≈ 0.69.

After one update, suppose the policy gives the preferred reply a log-ratio of +0.5 against the reference, and the rejected reply −0.5. With β = 0.1, the bracket is 0.1 × (0.5 − (−0.5)) = 0.1. The loss drops to −log σ(0.1) ≈ 0.64.

So the loss shrinks as the policy moves toward the preferred reply and away from the rejected one, relative to where it started.

Offline or online

DPO is offline. It trains on a fixed set of pairs. That makes it stable, cheap and reproducible. The cost is coverage. If the policy drifts into regions with no preference data, its behavior can be odd, because nothing corrects it there.

Online RL, such as PPO or GRPO, samples from the current policy, so it explores. It needs a live reward source and more infrastructure.

Practical tips

• Keep the reference frozen. It is usually the supervised checkpoint, and the policy starts from the same one.
• Tune β. If the model collapses, raise it. If it barely moves, lower it.
• Filter ties and noisy labels before training.
• Judge by win rate against the supervised model on a held-out set, not by the training loss.

Common mistakes

• Starting from a different checkpoint than the reference. The loss compares the policy to the reference, so the two must start together.
• Trusting the training loss. Loss can fall while the outputs get worse. Measure win rate.
• Treating DPO as a safety guarantee. It moves the model toward the labels. The quality of the labels sets the ceiling.

Check yourself

1. Why is the loss log 2 before any training? Answer: the policy and the reference agree, so the bracket is zero and −log σ(0) = log 2.
2. What does a large β do in DPO? Answer: it makes the loss sharply sensitive to the policy's drift from the reference, so the policy stays close to it.

Key takeaways

• DPO optimizes the KL-constrained RLHF goal with one loss on preference pairs.
• The loss raises the preferred reply's likelihood and lowers the rejected reply's, both relative to a frozen reference.
• It is offline. Online RL explores better but needs more infrastructure.
• Judge by win rate, and check that the reference matches the starting checkpoint.

Further learning

The DPO paper introduces the method:

Direct Preference OptimizationWhile large-scale unsupervised language models (LMs) learn broad world knowledge and some reasoning skills, achieving precise control of their behavior is difficult due to the completely unsupervised nature of their t…arXiv.org

For feedback that does not need humans for every label, see RLAIF and Constitutional AI.