Eval graders

A grader is the rule that decides whether one output passes. Graders range from a string comparison to a human reading the answer. Choose the cheapest grader that catches the failure you care about. Code is free and deterministic. A human is slow and expensive.

The grader ladder

The default order is code, then similarity, then judge, then human.

Walking through a harness

This harness runs three cases and reports a pass rate for each category. The fake model stands in for a real call.

def run_eval(cases, generate):
    results = {}
    for name, prompt, check, category in cases:
        out = generate(prompt)
        results.setdefault(category, []).append(check(out))
        if not check(out):
            print(f"FAIL {name}: {out}")
    return {cat: sum(oks) / len(oks) for cat, oks in results.items()}

def fake_model(prompt):
    if "refund" in prompt:
        return '{"tool": "refund", "order_id": "881"}' if "881" in prompt else '{"tool": "refund"}'
    return "Your order ships Monday."

cases = [
    ("r1", "refund order 881", lambda o: '"order_id"' in o, "tool_use"),
    ("r2", "refund my order", lambda o: '"order_id"' in o, "tool_use"),
    ("s1", "when does it ship?", lambda o: "Monday" in o, "faq"),
]
print(run_eval(cases, fake_model))

Output:

FAIL r2: {"tool": "refund"}
{'tool_use': 0.5, 'faq': 1.0}

The tool_use category passes half its cases. The check is a plain string test, so it costs nothing to run on every change.

Report the rate per category, not only the overall number. A 2-point overall gain can hide a 15-point drop in tool_use.

Write the expected behavior first

Every case needs a clear expected behavior: an exact string, the facts that must appear, a schema, or a short rubric. If the expected outcome is vague, the scores are noisy, and a failing case may not be a real failure.

Common mistakes

• Using a judge for a rule code can check. An assert on a field costs nothing and never disagrees with itself.
• Reporting one overall pass rate. It hides which category broke.
• Vague expected outputs. You cannot grade what you have not defined.

Check yourself

1. The agent must return a refund call with a valid order_id. Which grader do you use? A code check. Parse the tool arguments and assert the field exists.
2. Why not score every answer with a judge? Judges cost money, add their own errors and need calibration. Code is free and deterministic.

Key takeaways

• Pick the cheapest grader that reliably catches the failure.
• Use code for anything with an objective rule.
• Use judges only for judgment calls, and calibrate them first.
• Report pass rates for each category.

Further learning

Next, see how these graders run in CI and in production in offline and online evals. For judges specifically, read LLM-as-a-judge.