Guide

Production and case studies

Support evals and cost

Label 200 to 500 real tickets. Gate on policy and never-call cases. The cost model only counts if CSAT holds.

144 of 155 · 2 min read

Build evals from real tickets (privacy-reviewed). You are not measuring fluency. You are measuring path, policy and money.

Concept

Before you start: requirements already defined contained vs assisted vs escalated. The eval set is how you refuse to ship a prompt that breaks those definitions.

From historical tickets

  1. Sample 200–500 tickets stratified by category.
  2. Label expected path: RAG answer / tool / escalate.
  3. For RAG: required facts, forbidden claims, citation must support the answer.
  4. For tools: exact tool sequence or allowed set; never-call list for unsafe tools.
  5. For escalate: must-escalate cases (legal, self-harm, VIP).

Split the set into smoke (fast CI), full (pre-release), and safety (must stay perfect). Safety cases are never "mostly pass." Include adversarial tickets with injection strings and social-engineering attempts to grant refunds.

Metrics:

  • Deflection / autonomous resolution rate on the in-scope set
  • Policy violation rate (must be ~0 on the safety set)
  • Citation faithfulness (judge or human)
  • Handoff quality (human rates the context pack)
  • Router accuracy — wrong route is often the whole bug
  • Online: CSAT, reopen rate, median handle time for mixed human+bot

Run the suite in CI on prompt and retrieval changes. Add every production incident as a regression case within a week. When retrieval indexes change, re-score faithfulness even if prompts did not — stale chunks cause quiet regressions.

Walking through the annual sketch

Baseline human-only: 1M tickets/year, $4 fully loaded / ticket → $4M. Bot handles 60% at $0.04 model+infra / ticket and humans handle 40% at $4:

0.6M × $0.04  = $24,000
0.4M × $4     = $1,600,000
total         ≈ $1.62M
saved         ≈ $2.4M / year before building cost

Only if CSAT and reopen rates hold. If reopen rate rises 5 points, much of the savings evaporate in repeat contacts.

Cost drivers: tokens per turn, turns per ticket, cascade/routing mix, retrieval, human review. Cache the help-center prefix, route pure FAQ to a small model, batch offline evals. See latency and cost and caching.

Build a spreadsheet before launch: expected mix of FAQ vs tool vs escalate, tokens per path, model prices, human cost per ticket, target contained rate. Sensitivity-analyze ±20% on escalate rate — that variable usually dominates savings more than a 10% token trim.

Common mistakes

  • Safety cases that "mostly pass." They must stay perfect.
  • Not re-scoring after an index change. Stale chunks regress faithfulness with no prompt diff.
  • A cost model that ignores reopen. Repeat contacts eat the $2.4M.

Check yourself

  1. A prompt change is clean but the help-center index shipped new chunks. What do you re-run? Citation faithfulness. Stale or swapped chunks regress answers without a prompt diff.
  2. Which variable usually dominates savings more than a 10% token trim? Escalate rate. Sensitivity-analyze ±20% on it.

Key takeaways

  • Evals from historical tickets, with policy and never-call cases.
  • Split smoke / full / safety. Safety is binary.
  • Track deflection and CSAT. Reopen eats savings.
  • Escalate rate moves the bill more than a small token trim.

Further learning

Your AI Product Needs Evals (Hamel Husain)How to construct domain-specific LLM evaluation systems.Hamel’s Blog
AI Evaluations Clearly Explained in 50 Minutes (Real Example) | Hamel HusainPeter Yang

Next, how it reaches production: support rollout.

Case study: customer support agent

End-to-end: RAG over help docs, account tools, escalation to humans, evals and cost.

Evaluating LLM apps

A fixed set of inputs plus a way to score outputs, so you can tell whether a change helped.

Latency and cost optimization

Streaming, routing, cascades, batching and parallelism: making LLM apps fast and affordable without dropping quality.