Guide

Production and case studies

Support agent architecture

A router, a read-only RAG answerer, a bounded tool agent and a handoff pack. Not one free-roaming agent on day one.

142 of 155 · 2 min read

Keep the system a workflow with an agentic pocket, not a free-roaming agent on day one. Free-roaming demos look impressive in week one and create audit nightmares in week six when nobody can explain why a refund tool fired. Start narrow. Widen tools only when eval slices for that category stay green.

Concept

Before you start: RAG retrieves help-center chunks. Tool use is a bounded loop your runtime executes. The router decides which of those pockets runs.

Four components

  1. Router — small model or classifier: FAQ, account mutation, billing dispute, escalate-now. Cheap and evaluated hard.
  2. RAG answerer — retrieve help-center chunks, generate a grounded reply with citations. No account-mutating tools.
  3. Tool agent — bounded tools for order status, password reset links, refund proposals.
  4. Handoff — packages transcript, retrieval, tools tried, and a suggested reply for a human.

Trade-off

A single mega-agent with every tool is simpler to demo and harder to secure. Split FAQ (read-only RAG) from account actions so a prompt-injection in a ticket cannot easily reach refund tools. See guardrails.

Knowledge freshness

Sources: help-center articles (canonical), product release notes, curated snippets from resolved tickets (carefully reviewed), and policy PDFs with owners and expiry dates.

crawl / export → clean HTML → chunk (by heading) → embed → index
                     ↘ store article_id, url, updated_at, product_area

Every chunk carries updated_at and doc_version. Re-index on publish. For high-churn policies (pricing, refund windows), prefer shorter TTLs and a daily diff job that fails the deploy if critical articles are missing. Cite title + url. Prefer the newest chunk when versions conflict. If retrieval scores are weak, refuse and escalate rather than improvise.

Prompts stay split

The RAG answerer encodes product law, not vibes:

You answer only from <sources>. If sources lack the answer, say you don't know
and offer handoff. Quote policy thresholds exactly. Never invent SKUs, prices,
or refund windows. Cite source titles used. Do not call account tools.

The tool agent gets a different prompt: identity binding, confirmation rules, escalation triggers (legal language, self-harm, repeated frustration, VIP flag). Keep these prompts versioned separately so a FAQ tone tweak cannot loosen refund rules.

Wrong route is often the whole bug. Evaluate the router as its own slice.

Common mistakes

  • One agent with every tool on day one.
  • Improvising when retrieval is weak instead of handing off.
  • One shared prompt so a tone edit loosens refund rules.

Check yourself

  1. Why does the RAG answerer have no mutating tools? A prompt-injection in a how-to ticket then cannot reach refunds.
  2. Retrieval scores are weak. What does the answerer do? Refuse and escalate. Do not invent policy.

Key takeaways

  • Prefer router + RAG + bounded tool agent over one unrestricted agent.
  • Ingest help content with versions and citations. Refuse when retrieval is weak.
  • Version the FAQ prompt and the tool prompt separately.
  • Evaluate the router on its own. A wrong route is the whole failure.

Further learning

Next, the dangerous pocket: support tools and permissions.

Case study: customer support agent

End-to-end: RAG over help docs, account tools, escalation to humans, evals and cost.

Retrieval-augmented generation

The core RAG pipeline: ingest, chunk, embed, retrieve, generate with citations.

Tool use and function calling

Letting models call your code: tool schemas, the tool loop, and designing tools models use well.