Durable workflows

A single job is easy: send one email, resize one image. A process with several steps is harder. Place an order means charge the card, reserve stock, create a shipment and send an email. If step 3 fails, step 1 must be undone. If the server crashes in the middle, the process must resume, not restart from scratch.

The big idea

You can chain queue jobs. Step 1 finishes and publishes a message for step 2, and so on. This works, but the state of the process ends up spread across queues, database flags and logs. Nobody can answer "where is order 881?" without searching all three.

A durable workflow engine solves this. You write the process as ordinary code. The engine records the result of each step. If the process crashes, the engine replays the code and skips the steps that already finished.

Walking through an order

Here is a sketch of a Temporal workflow in TypeScript. Each activity is retried by the engine, and the result is saved.

import { proxyActivities } from "@temporalio/workflow"
import type * as acts from "./activities"

const { chargeCard, reserveStock, refundCard, createShipment } = proxyActivities<typeof acts>({
  startToCloseTimeout: "30 seconds",
  retry: { maximumAttempts: 5, backoffCoefficient: 2 },
})

export async function placeOrder(order: Order): Promise<void> {
  const charge = await chargeCard(order.id, order.total)
  try {
    await reserveStock(order.id, order.items)
  } catch (err) {
    await refundCard(charge.id) // compensation: undo the earlier step
    throw err
  }
  await createShipment(order.id)
}

Read it top to bottom:

1. The card is charged. The engine saves the charge result.
2. Stock reservation fails. The workflow runs the compensation, a refund, which undoes step 1.
3. If the server crashes after step 1, the engine resumes here and does not charge again.

This pattern is a saga: a sequence of steps, each with an undo.

When to use which

Job queues add retries, delays, priorities and a dashboard. They are the right tool for single jobs, and most application teams use them before they need a workflow engine.

Common mistakes

• Using a queue for multi-step state. Chained jobs make failures hard to trace. Use a workflow when order matters across steps.
• Forgetting compensation. If you can't undo a step, say so in the design, and put the risky step last.
• Non-idempotent activities. The engine will retry them. A charge that isn't idempotent can charge twice.

Key takeaways

• A workflow engine records each step's result, so a crash resumes the process instead of restarting it.
• Use an undo step (compensation) when a later step fails.
• Single jobs belong in a job queue. Multi-step processes with undo belong in a workflow engine.

Further learning

The Temporal documentation has tutorials for workflows, activities and retries.