Judge biases

The MT-Bench / LMSYS line of work, and a lot of follow-on, documented systematic judge biases. If you do not design around them, a "quality win" can be "the new answer is longer" or "it came first."

The usual biases

Reference answers help. When you have a gold reply, ask "is the candidate at least as correct as the reference on facts?" rather than open-ended "which is better?" That reduces style games.

Failure modes of the judge itself

• Rubric leakage. The judge rewards phrases that appear in the rubric examples, so models learn to parrot judge-bait. Keep exemplars diverse. Refresh them.
• Distribution shift. You calibrated on English support tickets. The app launches in Spanish. Re-label a slice.
• Provider silent updates. Judge model weights change behind an alias. Pin versions or re-calibrate on a schedule.
• Circular evals. You optimize the actor prompt against a frozen judge until it games the judge. Hold out a human-labeled set the actor never trains against, and spot-check with humans monthly.

Walking through a verbosity trap

Two answers to the same refund question. Answer A is 40 words, names the order id, and is correct. Answer B is 220 words, apologizes at length, and is also correct. An uncalibrated 1–10 judge gives B an 8 and A a 6. Your product wanted A. The rubric never said brevity is fine, so the judge used length as a quality proxy.

Add "Length is not a criterion. Brevity is fine." and two short PASS examples. Re-score the calibration set. If B still wins on facts, that is a real preference. If A and B now tie on faithfulness, the verbosity bias was the whole delta.

Common mistakes

• Pairwise without swapping order. Position bias decides the winner.
• Same-family judge and actor as the only signal for a launch claim.
• Never refreshing exemplars. The actor learns the judge-bait.

Check yourself

1. In pairwise mode, A wins when listed first and loses when listed second. What is that? Position bias. Count it as a tie, or average both orderings.
2. Why not use the same model family as the only judge of its own outputs? Self-preference. Use a different family when the claim matters, and calibrate against humans.

Key takeaways

• Position, verbosity, self-preference and style biases are documented, not theoretical.
• Swap pairwise order. Say length is not a criterion.
• Pin the judge model. Re-calibrate on shift.
• Hold out a human-labeled set so the actor cannot game the judge.

Further learning

The paper that measured human agreement and made judges mainstream:

Judging LLM-as-a-Judge with MT-Bench and Chatbot ArenaEvaluating large language model (LLM) based chat assistants is challenging due to their broad capabilities and the inadequacy of existing benchmarks in measuring human preferences. To address this, we explore using st…arXiv.org

Next, a protocol that swaps order on purpose: pairwise judge comparisons.