Production and case studies
Support agent requirements
Write autonomy, safety, handoff, freshness, latency and cost as constraints. Define in-scope and what "resolved" means.
Customer support is the default proving ground for LLM agents: high volume, messy language, a knowledge base you already have, and a clear economic story (deflect tickets without wrecking CSAT). It is also where unsafe tool use and confident wrong policies hurt real people.
Write requirements as product and safety constraints, not "use GPT." The target in this case study is answer ~60% of in-scope tickets autonomously, never take unsafe account actions, and hand off gracefully.
The constraint table
| Requirement | Why |
|---|---|
| Autonomous resolution ≥ 60% of in-scope tickets | Economic case vs human cost |
| Zero unauthorized refunds / PII leaks / policy violations | Trust and compliance |
| Graceful handoff with full context | CSAT when the bot fails |
| Answers grounded in current help center | Hallucinated policy is worse than no bot |
| p95 first-response latency under ~5 s with streaming | Feels usable in chat |
| Cost per autonomous resolution ≪ human handle cost | Unit economics |
Define in-scope: password resets and "where's my order?" may be in; medical advice and legal threats may always escalate. Scope is a product decision written as an allowlist of intents, not a hope that the model will "know." Revisit scope quarterly as the help center and tool catalog grow.
Important
"60% autonomous" is not "60% of messages get a model reply." Count resolved without human, with a clear definition (customer confirms / no reopen in 48h / CSAT survey). Publish the definition next to the metric so leadership cannot redefine success after the fact.
What "resolved" means in data
Instrument outcomes carefully or the 60% target becomes political.
- Contained: customer never reaches a human; issue not reopened within 48 hours.
- Assisted: human sends mostly bot draft (<20% edit distance).
- Escalated: human owns the thread after bot handoff.
- Abandoned: customer leaves mid-bot — investigate UX and latency, not only model quality.
Report contained rate on in-scope volume separately from "bot touched the ticket." Finance cares about contained. Engineering also cares about assisted (it still saves handle time). Pair every efficiency chart with CSAT and reopen rate so nobody ships a deflection that creates two follow-ups.
The economic case is the same arithmetic as support evals and cost: 1M tickets at $4 is $4M; 60% contained at $0.04 plus 40% human is about $1.62M, roughly $2.4M saved — only if CSAT and reopen hold. If reopen rises 5 points, repeat contacts eat the savings.
Common mistakes
- Counting model replies as resolutions. Count contained tickets.
- No allowlist of intents. The model then "decides" medical advice is in scope.
- Efficiency charts without CSAT and reopen. Deflection that creates two follow-ups is not savings.
Check yourself
- A ticket got three bot replies and then a human. Is it contained? No. Contained means no human and no reopen within 48 hours.
- Why publish the resolution definition next to the 60% number? So nobody redefines success after the fact.
Key takeaways
- Spec autonomy %, safety, handoff, freshness, latency and cost up front.
- In-scope is an allowlist, not a vibe.
- Contained ≠ bot-touched. Pair deflection with CSAT and reopen.
- The $2.4M sketch only counts if quality metrics survive contact with users.
Further learning
AI product UX for trust and handoff. Next, the shape of the system: support architecture.
Related
Case study: customer support agent
End-to-end: RAG over help docs, account tools, escalation to humans, evals and cost.
Evaluating LLM apps
A fixed set of inputs plus a way to score outputs, so you can tell whether a change helped.
Designing AI product experiences
Interaction design for probabilistic systems: trust, control, failure states and human-in-the-loop.