Pairwise judge comparisons

When comparing prompt A vs prompt B, or model A vs B, pairwise often matches human preference better than scoring each answer alone on a 1–10 scale. Pointwise scores drift: today's 7 and yesterday's 7 are not the same ruler. A forced choice on the same input is a stabler question.

The catch is position bias: the judge may prefer whichever answer you listed first. The protocol below cancels that.

The protocol

1. Same input → generate answer A and answer B.
2. Ask the judge which is better on one criterion (or overall, if you must).
3. Run again with A and B swapped.
4. Tally: A wins both → A. B wins both → B. Split → tie.

def pairwise_winner(prefer_first: bool, prefer_first_swapped: bool) -> str:
    # prefer_first True means the first argument won in that ordering
    a_wins = prefer_first and (not prefer_first_swapped)
    b_wins = (not prefer_first) and prefer_first_swapped
    if a_wins:
        return "A"
    if b_wins:
        return "B"
    return "tie"

print(pairwise_winner(True, False))   # A first, B first after swap → A
print(pairwise_winner(True, True))    # first slot wins both times → tie
print(pairwise_winner(False, True))   # B, then A after swap → B

Output:

A
tie
B

Report win rate with ties, and confidence intervals when n is small. The same statistical humility as in eval noise and experimentation pitfalls.

40 pairs,  22 A wins, 14 B wins, 4 ties
  A win rate among decisive = 22 / 36 ≈ 61%
  but n is small: a few flips change the story

400 pairs, 220 A, 140 B, 40 ties
  A win rate among decisive = 220 / 360 ≈ 61%
  now the same 61% is evidence, not a mood

A 55% win rate on 40 pairs is weak. On 400 pairs it is interesting. Pairwise doubles judge calls (two orderings). Cache identical (judge_version, A, B, order) so a re-run does not double the bill again.

Common mistakes

• Skipping the swap. Position bias picks the winner.
• Celebrating 55% of 40 pairs. That is noise-sized.
• Several criteria in one pairwise prompt. You will not know why A won.

Check yourself

1. A wins when listed first and B wins when listed first. What do you record? A tie. The orders did not agree.
2. Why is 55% of 40 pairs weak evidence? Small n. The same humility as a 3-point pass-rate move on a small suite.

Key takeaways

• Pairwise matches human preference better than two independent 1–10 scores.
• Always swap order. Only consistent winners count.
• Report ties. Do not hide them in the win rate.
• Need more pairs than you think. 40 is weak. 400 is interesting.

Further learning

Back to LLM-as-a-judge. For sample-size intuition, see eval noise.