Serverless event sources and retries

Serverless works best when something else produces the work. A queue message, an object upload, a schedule or a webhook each triggers a function. Those triggers retry on failure, so the same event can arrive more than once. Design every handler for that from the start.

Event sources

For SQS and Lambda, a common rule of thumb is a visibility timeout of at least six times the function timeout. A message that reappears while the first worker is still running it is processed twice.

Walking through a fan-out

One upload can trigger a chain. An S3 PutObject starts a function. It writes a message to a queue. The queue fans out to many workers. The workers write results to a warehouse. You never provisioned a fleet for the rare spike, but you did create new failure modes: partial success, duplicates and poison messages. Each needs a dead-letter queue (DLQ), a replay tool, or an idempotent write.

An idempotent handler is one where a repeat has no extra effect. This sketch uses an in-memory dict as a stand-in for a database table with a unique key:

seen = {}  # stands in for a table with a unique key on order_id

def handle(event):
    order_id = event["order_id"]  # a business key, not a random ID
    if order_id in seen:
        return "duplicate, skipped"
    seen[order_id] = "charged"
    return "charged"

event = {"order_id": "ord-881"}
print(handle(event))
print(handle(event))

charged
duplicate, skipped

In production, store the key in a database row with a unique constraint. Memory is lost when the environment is recycled.

Common mistakes

• Assuming exactly-once. Retries happen. Make the write safe to repeat.
• Doing heavy work inside a webhook. Reply quickly and put the work on a queue.
• Letting scheduled runs overlap. If a run takes longer than the interval, runs pile up.

Check yourself

1. A handler charges a card, and the platform retries the event. What happens without idempotency? Answer: the card is charged twice.
2. What does a visibility timeout of at least six times the function timeout protect against? Answer: a message reappearing for another worker while the first worker is still processing it.

Key takeaways

• Serverless events are at-least-once. Plan for duplicates.
• Make writes idempotent with a business key, such as an order ID.
• Send messages that keep failing to a dead-letter queue.
• Next, concurrency and connections explains a limit that event spikes hit first.

Further learning

Designing robust and predictable APIs with idempotency (Stripe)Online payment processing for internet businesses. Stripe is a suite of payment APIs that powers commerce for businesses of all sizes.stripe.com