Judge design

Three design choices dominate whether a judge is usable: the scale, the scope and the shape of the output. Get these wrong and calibration cannot save you.

Binary pass/fail with a critique beats 1–10

Ask the judge to explain briefly, then emit PASS or FAIL (or a 1–3 ordinal). Large numeric scales invite false precision: 7 vs 8 is noise, and humans disagree on them too. Binary forces a decision and makes precision and recall against human labels easy to compute.

You grade whether the ASSISTANT answer is faithful to SOURCES.

Rules:
- FAIL if any factual claim is not supported by SOURCES.
- FAIL if the answer invents a policy, price, or deadline.
- PASS if all claims are supported, even if the answer is incomplete.
- Length is not a criterion. Brevity is fine.

Write 2–4 sentences of reasoning, then a final line:
VERDICT: PASS
or
VERDICT: FAIL

SOURCES:
{{sources}}

QUESTION:
{{question}}

ASSISTANT:
{{answer}}

One criterion per judge

A single prompt that scores faithfulness and tone and completeness produces mushy labels and hard debugging. Split into specialist judges. A faithfulness fail and a tone pass tells you what to fix. A blended 6/10 does not.

When you have a gold answer, prefer reference-guided grading: "List factual differences from the reference; PASS if none are material." That reduces style bias versus open-ended quality scores.

For multi-requirement responses, a checklist judge that returns yes/no per bullet (addressed greeting, mentioned order id, offered escalation) often beats one holistic verdict. Each bullet can become its own metric on the eval dashboard.

Structure the output

Use structured outputs or a strict parseable trailer (VERDICT: PASS). Log the critique. When the judge is wrong, the critique shows whether the rubric or the model is the problem.

Common mistakes

• One judge for every quality. You cannot tell what to fix.
• A 1–10 scale with no critique. You get a number you cannot debug.
• Making length a hidden criterion. Verbose answers then win.

Check yourself

1. You care about faithfulness and tone. One judge or two? Two specialist judges. A fail on one and a pass on the other tells you what to fix.
2. Why prefer PASS/FAIL over 1–10? 7 vs 8 is noise. Binary is a decision you can compute precision and recall against.

Key takeaways

• Binary (or a tiny ordinal) plus a short critique beats a 1–10 score.
• One criterion per judge.
• Parse the verdict. Log the critique.
• Reference-guided and checklist judges reduce style games.

Further learning

Next, measure the judge against humans in calibrating LLM judges.