Calibrating LLM judges

A judge you have not compared to a human is a vibes function with an API bill. Calibration means measuring agreement on a held-out set, then adjusting the rubric (or the model) until agreement is good enough for your risk.

The loop

1. Collect 100 to 200 examples for one failure mode, labeled by a domain expert (PASS/FAIL).
2. Run the judge. Treat FAIL-detection as a classification problem: precision, recall, F1 for the FAIL class (or PASS, whichever is rarer and costlier to miss).
3. Read disagreements. Fix the rubric first (ambiguous rules, missing edge cases), then examples in the judge prompt, then consider a stronger judge model.
4. Freeze the judge prompt and version. Re-check calibration when you change the judge model or when the product distribution shifts.

Track judge quality next to product metrics. If judge–human agreement drifts from 85% to 70%, your "eval improvement" may be judge rot, not product gain.

Walking through 120 faithfulness cases

You label 120 cases: 40 FAIL, 80 PASS. Judge A catches 28 of 40 fails (recall 0.70) but also flags 15 of 80 passes.

FAIL recall     = 28 / 40 = 0.70
FAIL precision  = 28 / 43 ≈ 0.65

For a CI gate that blocks deploys, low precision means false alarms and alarm fatigue. Tighten the rubric ("incomplete is still PASS"). For a safety filter where missing a fail is expensive, push recall first and accept more human review on flagged cases.

Iterating on the judge prompt

Treat the judge like any other prompt under test.

1. Add 5 to 10 disagreement cases as few-shot exemplars (human label plus a short why).
2. Clarify edge rules ("quotations from SOURCES that are truncated still count as supported if the claim matches").
3. Lower temperature. Pin the judge model version.
4. Recompute metrics on the full calibration set, not only on the cases you just fixed. Overfitting judges is real.
5. If agreement plateaus below your bar (many teams aim for roughly human–human agreement, often about 70 to 85% depending on task hardness), split the criterion further or accept human review for that slice.

A second pass: the faithfulness judge misses "soft hallucinations," answers that rephrase a policy incorrectly without inventing numbers. You add a rubric line: "FAIL if the answer changes a condition, deadline, or eligibility rule relative to SOURCES." FAIL-recall rises from 0.70 to 0.86. Precision dips slightly. You accept the trade because a wrong policy is costlier than an extra false FAIL in CI.

Common mistakes

• Fixing only the cases you just read. Overfit. Re-score the full set.
• Swapping the judge model before fixing the rubric. Ambiguous rules will fail on the new model too.
• Never re-calibrating. Provider silent updates and language launches shift the distribution.

Check yourself

1. 40 real FAILs, judge catches 28 and also flags 15 passes. Precision on FAIL? 28 / 43 ≈ 0.65.
2. Agreement was 85% and is now 70% after a "quality win." What might have happened? Judge rot. Re-check the judge against humans before you trust the product score.

Key takeaways

• Label 100 to 200 examples per failure mode.
• Track precision and recall of the judge, not only product pass rate.
• Fix the rubric first, then few-shots, then the model.
• Freeze the judge version. Re-calibrate on model or distribution change.

Further learning

Next, the systematic ways judges go wrong: judge biases.