Online evals, feedback and alerts

Offline evals run before a change ships. Online evals run on live traffic after it ships. They catch the problems the golden set missed: new kinds of questions, changes in the provider's model and slow drift. Judging every request is too slow and too expensive, so online checks sample. Cheap checks run on everything. Expensive judges run on a small slice, weighted toward the cases that look bad.

The big idea

Use three layers, from cheapest to most expensive:

1. Code checks on every response. Is the schema valid? Does the answer contain a banned phrase? Does the cited URL exist?
2. Sampled LLM judges. Score faithfulness or policy compliance on a small slice of traffic.
3. Business metrics. Deflection rate, edit rate, escalation rate and customer satisfaction.

A smoke detector system is a fair picture. A cheap sensor checks everything, and a more careful inspection runs on the alarms. The analogy stops at one point: a judge can be wrong too, so you check its labels against humans on a regular basis.

Walking through the sampling math

Suppose 100,000 requests arrive in a day. Judging all of them costs too much. Sample 1%:

requests per day:          100,000
judged at 1% sample:         1,000 judge calls
thumbs-down traces (all):      500
oversampled thumbs-down:       500 (judged 100%)

Two rules shape the sample. Stratify by route, so FAQ, tool and escalation traffic each get a share. Oversample the negatives, since thumbs-down and escalated traces are where the real failures live. Cap the judge spend per day. If the judge provider goes down, fall back to code checks only.

Feedback becomes eval data

Capture signals that correlate with quality:

• Explicit: thumbs up or down, and a short reason from a small list, such as wrong_policy or hallucinated.
• Implicit: regenerations, edits to suggested text, quick abandonment, escalation to a human, and copied citations.

Send the negatives to an annotation queue. A person labels about 50 to 100 traces a week, groups the failure modes and adds the best examples to the golden set. That loop is what separates LLMOps from having a dashboard installed.

Keep judge prompts versioned with product prompts. Judges can drift too.

What should page someone

Alert on quality signals, not only on infrastructure:

• A sudden drop in thumbs-up or a rise in escalations.
• A spike in tool errors or empty retrievals.
• A jump in cost per request, which often means a prompt blowup or a cache miss storm.
• A judge score that falls below a rolling baseline.

Common mistakes

• Alerting only on latency and 5xx errors. Those miss a fluent wrong answer.
• Judging 100% of traffic. It is slow and costly. Sample, and oversample the failures.
• Trusting the judge blindly. Spot-check its labels against human labels every week.
• Using the judge as the only check. Code checks are cheaper and catch the simple failures first.

Check yourself

• Why oversample thumbs-down traces? They are where real failures concentrate, so they give the most labels per judge call.
• Why keep code checks on every response? They are cheap, so they cover all traffic, and they catch malformed output before any judge is needed.

Key takeaways

• Layer the checks: code on everything, judges on a sample, business metrics on the whole funnel.
• Turn thumbs-down, edits and escalations into labeled eval cases.
• Alert on quality metrics, not just latency and errors.
• Pin model versions, and spot-check the judge against humans.

Further learning

Next, see how gateways handle keys, quotas and spend in LLM gateways.