Durable agent execution

Agents routinely outlive a single HTTP request: research for 10 minutes, wait for a human overnight, retry a rate-limited model call, resume after a deploy. Durable execution (Temporal and similar workflow engines) records progress so a crash mid-flight resumes from the last completed step instead of starting over.

The big idea

• The workflow is deterministic orchestration code: "call model → maybe tool → maybe wait for signal."
• Activities are side effects: model API calls, sandbox commands, sending email. They retry with policy.
• Signals / updates wake a sleeping workflow (human approved, new user message).
• State is persisted by the engine. Workers are fungible.

When you need this: overnight coding agents, human-in-the-loop ops, multi-hour research, anything that must not double-charge a customer because a pod ran out of memory mid-tool. When you don't: sub-30-second chat turns with a few tool calls. A simple loop plus a job queue may be enough.

steps = ["plan", "research", "wait_for_approval", "synthesize"]
done = {"plan", "research"}  # already completed before the crash
for step in steps:
    if step in done:
        continue
    if step == "wait_for_approval":
        print("sleeping on stored state until signal")
        break
    print("run activity:", step)

Output:

sleeping on stored state until signal

The completed plan and research activities are not re-billed. The wait holds no app-server thread.

Activities retry. Model providers sometimes return 500 after they actually produced a completion you never saw. Design for at-least-once:

• Tool writes use idempotency keys.
• "Send email" stores message_id before send. Retries no-op if present.
• Sandbox bootstrap is create-or-reuse by run_id.

You will not get perfect exactly-once across LLM plus tools plus human. You can get safe retries.

See event-driven architectures for the messaging around jobs and notifications.

Common mistakes

• Holding an HTTP request open for an overnight approval. The process dies; the plot is lost.
• Retrying a charge without an idempotency key. The customer pays twice.
• Starting a second run when the user injects guidance. Append a message and wake the existing workflow.

Check yourself

1. Why not asyncio.sleep for three hours on the API box? The process can die. Durable state waits for free; a thread does not.
2. A model API returns 500 after it actually billed you. What must the activity do? Treat the call as at-least-once and key it so a retry does not double-charge.

Key takeaways

• Durable workflows survive crashes and long waits. Activities wrap side effects.
• Use them when a run outlives a request. Skip them for short chat turns.
• Design tool activities for at-least-once retries with idempotency keys.
• Cancellation must flip a flag the workflow checks, abort in-flight sandbox commands and stop spending.

Further learning

What you persist between steps is agent state.