Preference pairs and the Bradley-Terry reward model

Supervised fine-tuning teaches a model to imitate written demonstrations. Demonstrations are expensive to write well, though, and many goals are easier to express as a comparison than as a perfect answer. "Be concise but complete" is hard to demonstrate and easy to judge. So instead of writing answers, labelers compare two model replies and pick the better one. Those comparisons, called preference pairs, are the signal for the rest of the pipeline.

Why pairs beat demonstrations

• Judging is faster than writing. Ranking two samples takes less effort than composing a full answer.
• Pairs say what not to do. A demonstration shows only the good path. A rejected sample shows a failure, such as a refusal that should not have happened, or leaked reasoning.
• Pairs can push past the demonstrations. Optimizing against preferences can move the model beyond the examples it was shown. That helps, and it can also hurt.

The Bradley-Terry model

The Bradley-Terry model says that the chance a preferred reply beats a rejected one depends on the gap between their scores:

P(y_w ≻ y_l | x) = σ( r(x, y_w) − r(x, y_l) )

Here σ is the sigmoid and r is the reward model's score. A large gap gives a probability near 1. A zero gap gives 0.5.

Walking through one training step

The reward model gives the preferred reply a score of 1.2 and the rejected reply 0.1. The gap is 1.1:

P(preferred wins) = σ(1.1) ≈ 0.75
loss = −log(0.75)  ≈ 0.29

Training minimizes this loss over many pairs, which is ordinary binary classification on score differences. Gradient descent raises the preferred score and lowers the rejected one.

If the model already separates a pair well, the loss is near zero and the pair teaches little. Near-ties and subtle failures carry the most information. A batch of obvious wins adds little.

Collecting the data

The usual loop:

1. Sample prompts from real traffic, after privacy filtering, plus a set of hard cases.
2. Generate two or more completions from the current model or a pool of models.
3. Have annotators choose a winner, a loser or a tie, and flag safety issues.
4. Enforce a written guideline, such as "prefer correct over witty" and "refuse when appropriate."

Costs add up quickly. Expert labels can cost on the order of dollars per comparison, and a small open-source run may need tens of thousands of pairs. Measure inter-annotator agreement, since noisy labels train noisy reward models.

Common mistakes

• Skipping the rubric. Without written guidelines, labelers disagree, and the reward model learns the average of their habits.
• Ignoring ties. A tie is information. Dropping ties, or forcing a winner, can teach the wrong lesson.
• Scoring the reward model by its training loss alone. Check it against a held-out set of human preferences.

Check yourself

1. If the preferred score is 0.5 higher than the rejected score, what does Bradley-Terry predict? Answer: σ(0.5) ≈ 0.62, a 62% chance the preferred reply wins.
2. Why are near-ties more useful than obvious wins? Answer: the loss on an easy pair is already near zero, so the model learns little from it.

Key takeaways

• Comparing two replies is easier than writing a perfect one, so preference pairs scale better than demonstrations.
• Bradley-Terry turns a score gap into a win probability with a sigmoid.
• The reward model trains on pairs with a simple logistic loss.
• Data quality, guidelines and agreement matter more than clever loss tricks.

Further learning

Next, see how a reward model is used to train the policy with a KL leash in KL leash in RLHF. The section overview is RLHF and preference optimization.