Support agent rollout

Do not turn on refunds in week one. Each stage needs a go/no-go on the eval suite and a CSAT floor. Feature-flag by percentage of traffic and by locale so you can stop a bad language rollout without a global outage.

The stages

1. Shadow — bot drafts, human sends; compare acceptance.
2. FAQ-only — RAG live for how-to; all mutations escalate.
3. Read tools — order status live.
4. Write tools with confirm — refunds as proposals.
5. Expand categories as eval gates pass.

Kill switch: disable writes, then disable the bot entirely, without redeploying the whole support app. Practice the kill switch in staging once. An untested flag is not a safety control.

Tell customers they may be helped by an assistant, and always offer "talk to a human" that actually works. A hidden escape hatch destroys trust faster than a slightly dumber bot. Put that control in the same place on every channel.

Staffing the human side

Automation changes human work. Agent desktop requirements:

• One-click adopt or edit of the bot's draft.
• Visible retrieval chunks and tool results.
• Macros for common escalations.
• Feedback buttons that file eval labels (wrong_policy, missed_escalate, good_deflection).

Train humans that fixing the draft and tagging the failure mode is part of the job. That is how the bot improves.

Measure assisted handle time as carefully as deflection. If humans spend longer cleaning bad drafts than writing from scratch, the bot is negative leverage for that category — turn it off for that route until evals improve. Some teams start with "bot suggests macros" rather than full drafts to build trust with agents.

Channels and languages

Email, chat and social DMs have different length and tone norms. Route by channel: shorter grounded answers in chat, fuller citations in email. For multilingual support, retrieve in the customer's language when you have localized help centers. Do not translate policy loosely in the model if legal text must stay exact — link the canonical article instead. Voice/IVR handoffs need a short spoken summary plus the same context pack for the human who picks up.

Walking through a kill

A refund eval slice goes red after a prompt tweak. The flag order:

1. disable write tools     (propose_refund off, reads stay)
2. if CSAT drops, disable bot  (FAQ and tools off, human queue only)
3. no app redeploy

Practice step 1 in staging. If you cannot turn writes off in one flag, you do not have a kill switch.

Common mistakes

• Skipping shadow and shipping FAQ live on day one.
• A "talk to a human" button that loops back to the bot.
• No desktop labels, so production failures never become eval cases.

Check yourself

1. Writes are live and a safety slice goes red. What is the first flag? Disable write tools. Reads and FAQ can stay while you investigate.
2. Why tag missed_escalate on the desktop? So the next eval suite contains the failure, not only a cleaned-up reply.

Key takeaways

• Roll out shadow → FAQ → reads → confirmed writes.
• Keep a practiced kill switch. Flag by traffic and by locale.
• Offer a human escape that actually works.
• The desktop is part of the product. Labels from agents become eval cases.

Further learning

Patterns for Building LLM-based Systems & Products (Eugene Yan)Evals, RAG, fine-tuning, caching, guardrails, defensive UX, and collecting user feedback.eugeneyan.com

What We Learned from a Year of Building with LLMsA practical guide to building successful LLM products, covering the tactical, operational, and strategic.Applied LLMs