Offline and online evals

An offline eval scores the system on a fixed dataset before it ships. An online eval scores real traffic after it ships. Offline evals catch regressions you can predict. Online evals catch failures you never put in the set. You need both.

Two kinds, two jobs

Running offline evals in CI

Split the offline suite into three tiers, so cheap checks run often and slow ones run less often.

Stochastic checks can be flaky. Use temperature 0 where you can, average several trials, or use enough cases that a few points of noise is expected.

What online evals catch

Online evals find what the offline set missed. Sample production traces, put them into the same failure categories, and grade a slice with the same checks. Thumbs, edits and escalations are weak labels.

When an online failure is real, make it an offline case within a day. That is how the loop gets stronger over time.

Record every run

Store each eval run as a row with these fields: timestamp, git_sha, prompt_version, model_id, retrieval_config, suite_id, category, pass_rate, n, cost_usd and p50_latency_ms. A chart of pass rates means little unless you know which prompt and model produced it.

When you want to learn something, change one thing per experiment. If you change the model and the prompt together and the score moves, you cannot tell which change did it.

Common mistakes

• Treating thumbs as the KPI. They are noisy and they skew toward strong feelings.
• Changing two things at once. You lose the ability to attribute the change.
• Unversioned prompts. Without a version, a pass-rate chart is guesswork.

Check yourself

1. A new model passes the offline suite, but users complain about a topic the suite never covered. Which eval caught it, and what now? The online eval caught it. Sample those traces, label them into a category and add them as offline cases.
2. Why change one axis at a time? So you can tell which change moved the pass rate.

Key takeaways

• Offline suites guard against regressions. Online sampling finds what you did not test.
• Run a smoke suite in CI on every prompt or tool change.
• Store every run with its prompt, model and config versions.
• Feed every real failure back into the offline set.

Further learning

Next, learn why small suites give noisy numbers in eval noise and sample size. When you A/B two prompts in production, use the same care as in experimentation pitfalls.