Agents and tools
Agent infrastructure
Running agents for real: sandboxes, durable execution, state, permissions and long-running tasks.
A demo agent is a for loop in a request handler. A production agent survives deploys, waits hours for a human approval, runs untrusted code without torching the host, remembers artifacts across turns and leaves an audit trail when something goes wrong. That gap is agent infrastructure.
The model loop from agents stays the same. The runtime around it changes. Treat agent runtimes as a backend service — with SLOs, load shedding and dependency budgets — not as a clever script beside the monolith. An agent that "books travel" is an availability, correctness and abuse problem: retries must not double-book, prompts must not exfiltrate other tenants' itineraries, and a stuck run must be visible to on-call.
Concepts in this section
Read them in order. Each one builds on the one before.
| Step | Concept | What it answers |
|---|---|---|
| 1 | Sandboxes | Where does untrusted code actually run? |
| 2 | Durable execution | How does a run survive a crash or an overnight wait? |
| 3 | State and memory | What do you persist, and what does the model reload? |
| 4 | Permissions and audit | Who is allowed to refund, and can you prove it? |
Build in this order when time is limited: authz and audit first (prevent irreversible harm), then budgets and timeouts (financial harm), then sandbox (host compromise), then durable runs and progress UX (operational flakiness), then fancy memory. Skipping ahead to memory while writes are unauthenticated is a common failure.
Long runs as product surface
Accept the job quickly. Return a run id. Stream or poll progress ("searching docs…", "waiting for approval…"). Notify on completion. Let the user cancel, pause or inject guidance mid-flight. Show intermediate artifacts, not only the final blob.
Silence for five minutes feels broken even if the agent is working. Cancellation must be real: flip a flag the workflow checks, abort in-flight sandbox commands and stop spending. A Cancel button that only hides the UI while the agent keeps refunding is a trust-destroying bug.
Cost and rate limits for fleets
One chatty agent is a bill. A fleet is an outage waiting to happen.
- Per-run budgets (tokens, dollars, step count) enforced in the runtime, not "please" in the system prompt
- Per-tenant and global rate limits; queue and shed load
- Cache identical retrievals and prompt prefixes (LLM caching)
- Tier models: cheap workers, expensive orchestrator
- Kill switches: disable tool classes or entire agent products when error rates spike
A fleet of 200 concurrent research agents in the source hit provider RPM limits at 09:00 UTC. Global 429s cascaded into Temporal retries that amplified traffic. The fix: per-tenant token buckets, jittered retries and a shed-load mode that paused new runs while draining.
Trade-off
Shared provider keys with only global limits are simple and unfair. One noisy tenant starves others. Prefer per-tenant quotas once you have more than a handful of customers.
Common mistakes
- Trust before cleverness is inverted. Fancy memory, unauthenticated writes.
- Global 429 retries that stampede the provider.
- A progress UI with no matching cancel.
Key takeaways
- Sandbox code and browsers. Durable workflows for long waits. Persist state aggressively.
- Authorize tools against real identity. Audit every side effect.
- Background UX (progress, notify, cancel) is part of the runtime.
- Enforce per-run and per-tenant budgets before the finance email arrives.
- Design activities for at-least-once retries with idempotency keys.
Further learning
Temporal documentationTemporal Platform Documentationdocs.temporal.io Effective context engineering for AI agents (Anthropic)Anthropic is an AI safety and research company that's working to build reliable, interpretable, and steerable AI systems.anthropic.com
Start with sandboxes.
Related
Agents and agent architectures
Workflows vs agents, the core patterns, ReAct loops, planning and knowing when to stop.
Event-driven architecture, CQRS and event sourcing
Building systems around events, from change data capture to event sourcing and read models, and knowing when it simplifies a system versus creating a distributed mess.
IAM and least privilege
How cloud identity works, why least privilege is a habit rather than a checkbox, and where the org-level guardrails sit. Start here for the map of IAM pages.