Guardrails, security and safety

LLM apps mix trusted instructions (your system prompt, tool policies) with untrusted text (users, uploaded PDFs, web pages, tool results, email). The model cannot reliably tell those apart the way a SQL engine distinguishes code from parameters. That is how next-token prediction works, not a temporary bug.

Security is therefore mostly architecture: what the model is allowed to see, what tools it can call, and where data can leave. It is not a magic classifier that "detects injection."

Treat the model as a confused deputy: persuasive, obedient and unable to hold a hard trust boundary inside the prompt.

The threat model

OWASP's Top 10 for LLM Applications catalogs these. Use it as a checklist in a design review, then map each item to a control in your architecture, not a generic "add a guardrail API."

Concepts in this section

Pair this with MCP if you expose tools, and with secrets handling for anything the agent can touch.

Hallucination is a safety issue too

Attackers are not the only harm. Confident wrong answers harm users.

• Ground answers in retrieved docs. Instruct abstention when evidence is missing.
• Require citations and verify them in code when you can (the span must appear in the sources).
• Prefer structured extraction with validation over free-form prose for high-stakes fields.
• Measure faithfulness with evals and judges. Treat regressions as release blockers for RAG apps.

Abstention is a feature. "I don't know — escalate?" beats an invented policy number.

Common mistakes

• Prompt-only security. "You are a secure agent that never leaks data" is not a boundary. Alignment reduces accidents. It does not stop a well-crafted injection against a tool-rich agent.
• Stacking filters instead of breaking a capability. One clean break in the trifecta beats three half-filters.
• Authorization in the prompt. The model may request get_account(id=...). Your tool code must check that id belongs to the session user.
• "We'll add Llama Guard" as the only mitigation when send-email is in scope.

Key takeaways

• Prompt injection is inherent to mixing instructions and data. Mitigate impact with architecture.
• The lethal trifecta is private data plus untrusted content plus an exfiltration channel. Break one leg.
• Prefer least-privilege tools, allowlisted egress, human confirmation and quarantine over prompt-only defenses.
• Treat traces as sensitive data. Redact, retain on purpose, and know provider terms.
• Red team for exfiltration paths whenever tools or data sources expand.

Further learning

The clearest statement of agent exfiltration risk.

The lethal trifecta for AI agents (Simon Willison)If you are a user of LLM systems that use tools (you can call them “AI agents” if you like) it is critically important that you understand the risk of …Simon Willison’s Weblog OWASP Top 10 for LLM ApplicationsOWASP Top 10 for Large Language Model Applications - An OWASP lab projectOWASP Foundation Llama Guard (arXiv)We introduce Llama Guard, an LLM-based input-output safeguard model geared towards Human-AI conversation use cases. Our model incorporates a safety risk taxonomy, a valuable tool for categorizing a specific set of saf…arXiv.org