Safety controls

Defense in depth. Prefer controls that still work when prompt injection succeeds. Asking the model to behave is the last layer, not the first.

Controls that work

Least-privilege tools. Narrow schemas. Separate read tools from write tools. Do not give a support bot raw SQL. Scope OAuth tokens to the minimum. For MCP servers, treat each server as code execution with its own trust tier.

Human confirmation. Irreversible or external actions (refunds, sends, deletes, public posts) need an explicit user or operator click. The model proposes. A human dispatches.

Allowlisted egress. Outbound HTTP only to known hosts. Block arbitrary fetch(url) with query-string exfiltration. Strip or rewrite markdown links that phone home with secrets in the URL.

Output filtering. Scan responses for secret patterns (API keys, connection strings) before display or before tool args execute. Imperfect, but it catches clumsy leaks.

Dual-LLM / quarantine. One model handles untrusted content in a sandbox with no private-data tools. It emits a structured summary. A second privileged model sees only the sanitized summary plus private tools. Untrusted text never shares a context window with exfiltration-capable tools.

Capability tokens. Some systems compile the model's plan into a capability-limited program (only certain APIs, row-level filters). Harder to build. Much stronger than "please don't."

Authorization belongs in your service code. The model may request get_account(id=...). The tool must enforce that id belongs to the session user. Models are not access-control systems.

Mapping to the usual risks

Content safety and PII

Jailbreaks try to bypass safety training or your policy (roleplay, encoded payloads). Provider models ship with refusal behavior from preference training. You still need app-level policy for brand and legal.

A practical stack: cheap input classifiers for obvious abuse, a moderation model on input and output, a short policy in the system prompt, and a refusal UX that says what you can do next. Measure over-refusal with evals.

LLM traces are a goldmine for debugging and a liability for privacy. Redact emails, phone numbers and card-like strings. Encrypt traces at rest. Define retention. Know your subprocessors: training opt-out, zero-retention APIs, regional endpoints. Never paste secrets into prompts when a tool could fetch them server-side under auth. See secrets and encryption.

Common mistakes

• Relying on the system prompt as the security boundary. Alignment is not a lock.
• Giving the model raw SQL or a broad OAuth token. Privilege belongs in the tool.
• Logging full prompts forever in plaintext. Traces contain PII.

Check yourself

1. A support bot can refund. Where does the "are you sure?" click belong? In your UI, before the refund tool runs. The model proposes. A human dispatches.
2. Why run untrusted PDFs through a sandbox model with no private-data tools? So injected instructions never share a window with tools that can send data out.

Key takeaways

• Enforce least privilege, confirmation, allowlisted egress and quarantine in code.
• Authorization is in the tool, not in the prompt.
• Content safety needs classifiers plus policy plus refusal UX, measured both ways.
• Treat traces as sensitive. Redact, encrypt, retain on purpose.

Further learning

Next, attack your own design in red teaming LLM apps. For classifier-style input and output filters:

Llama Guard (arXiv)We introduce Llama Guard, an LLM-based input-output safeguard model geared towards Human-AI conversation use cases. Our model incorporates a safety risk taxonomy, a valuable tool for categorizing a specific set of saf…arXiv.org