LLM-as-a-judge

Some qualities of an LLM app cannot be checked with assert. Faithfulness to sources, tone that matches brand, whether a support reply should have escalated: these need judgment. An LLM-as-a-judge asks another model to grade the output against a rubric.

Done well, it scales human taste. Done poorly, it is an expensive random number generator that rubber-stamps verbose nonsense.

Judges sit inside the broader eval loop. They are one assertion type among many, not a replacement for code checks, and never a replacement for reading traces.

When to use a judge vs code

Prefer code / schema	Prefer an LLM judge
JSON validity, required fields	Faithfulness to retrieved context
Exact IDs, enums, tool-arg types	Tone, empathy, brand voice
Unit tests on generated code	"Did the reply address the user's actual ask?"
Latency / cost thresholds	A rubric with several nuanced criteria
Regex for forbidden substrings	Comparing two free-form answers

Trade-off

If a failure mode has an objective rule, write code. Judges add cost, latency and their own error rate. Use them when a competent human would need to read and decide, and when you can afford to calibrate against human labels. Mixing both on the same case is normal: schema must pass and the faithfulness judge must pass.

Foundation-model labs popularized judges for open-ended chat quality (MT-Bench, Chatbot Arena-style pairwise). Product teams reuse the idea for application-specific rubrics, which are usually more valuable than generic "helpfulness 1–10."

A second use is pipeline grading: judge only retrieval faithfulness, or only whether a tool plan was justified, while code checks the JSON. Narrow scope makes calibration tractable.

Concepts in this section

Step	Concept	What it answers
1	Judge design	Binary, one criterion, a parseable verdict?
2	Calibrating LLM judges	How often does the judge agree with a human?
3	Judge biases	Why does it prefer the long, confident answer?
4	Pairwise judge comparisons	How do you A/B two prompts with a judge?

Cost control

Judges can dominate eval spend: every case × every specialist judge × retries × pairwise double calls.

Sample. Run expensive judges on a subset nightly. Run code checks on everything in CI.
Cascade. Code gates first. Only run the judge if schema and retrieval checks pass.
Smaller judge models. Once calibrated, many teams move from a frontier judge to a mid-size model and re-check agreement. If F1 holds, keep the savings.
Cache. Identical (judge_prompt_version, input, output) should not be re-graded.
Shorter critiques. "2 sentences max" cuts output tokens.

A 300-case suite with three specialist judges at about 1k tokens in and 200 out each is hundreds of thousands of tokens per full run. Fine weekly. Painful on every keystroke. Tier your suites: smoke vs full.

Worked sketch: n = 200 faithfulness cases, judge prompt ≈ 1,200 input tokens, critique ≈ 150 output. At mid-2020s API prices, one full run is a few cents to a couple of dollars depending on the judge model. Pairwise A/B doubles calls again. Cache aggressively. Grade diffs only when the actor output or judge version changes.

Common mistakes

A 1–10 helpfulness score as the only metric. 7 vs 8 is noise.
Letting the same model grade its own production outputs as the sole quality signal, without calibration.
Optimizing the actor prompt against a frozen judge until it games the judge. Hold out a human-labeled set.

Key takeaways

Use judges for subjective or multi-factor quality. Keep code checks for objective rules.
Prefer binary specialist judges with short critiques over one blended 1–10 score.
Calibrate against human labels. An uncalibrated judge is not an eval.
Control cost with cascades, sampling, smaller calibrated judges and caching.