Evaluating LLM apps

Without evals, every prompt change is a vibe check. You ship something that looked good on five examples, users hit the sixth, and you cannot tell whether last week's change helped. An eval is a fixed set of inputs plus a way to score outputs, run automatically so you can compare versions.

It is the test suite for an LLM app, except scores are rates, not binary pass/fail, and the system is stochastic. Prompting, RAG, tools and model choice are levers. The suite is how you know which lever moved the product.

Concept

An offline eval scores a fixed dataset before you ship. An online eval scores live traffic. You need both: offline for regressions, online for failures you never put in the set.

The loop

Error analysis comes first. The eval set is a consequence of failures you have seen, not a wishlist of vendor metrics.

A check for "refund tool called without an order ID" moves a product. A generic "helpfulness" score rarely does.

Concepts in this section

Read them in order. Each one builds on the one before.

Step	Concept	What it answers
1	Error analysis and failure taxonomies	What is actually going wrong?
2	Eval graders	What rule decides that one output passed?
3	Offline and online evals	What runs in CI, and what scores live traffic?
4	Eval noise and sample size	Is a 3-point gain real, or luck?

For subjective quality that code cannot check, use a calibrated LLM judge.

How big should the set be?

Start small and grow. Twenty cases that catch your top failure beat five hundred fluffy ones.

Purpose	Rough size
First error-analysis pass	About 50 to 100 traces
Code-based suite, per check	Often 20 to 100
Judge calibration, per failure mode	About 100 to 200 human-labeled examples
Full product offline suite	Often 100 to 500 or more, grown with coverage

Keep a held-out slice you do not look at while tuning. Write down the expected behavior per case: an exact string, required facts, a schema, or a short rubric.

A JSONL file and a script is enough to start. Platforms such as promptfoo, Braintrust and Langfuse help once many people touch the suite. If you cannot browse traces and annotate failures quickly, fix the data viewer before buying a judge dashboard.

Common mistakes

Skipping the taxonomy. Generic toxicity or helpfulness scores look scientific and rarely predict whether the refund tool still works.
Shipping on the overall number. A 2-point overall gain can hide a 15-point drop in tool_misuse.
Celebrating a 3-point move on 40 cases. That is often noise. See sample size.
Changing the model and the prompt together. You cannot tell which change moved the score.
Leaving prompts and eval sets unversioned. A pass-rate chart without a prompt and model id is folklore.

Key takeaways

Evals are the development loop. Vibes do not scale past a demo.
Error analysis on real traces creates the taxonomy; the taxonomy creates the evals.
Prefer code-based checks. Use judges only where judgment is required, and calibrate them.
Report pass rates by category. Watch cost and latency next to quality.
Offline suites catch regressions. Online sampling catches shift. Promote every real failure into the set.