Retries and dead-letter queues

A message fails. Maybe the database timed out, or the payment API returned an error. Should you retry? If so, how soon? And what if it fails every time? These three questions shape how a queue behaves under failure.

The big idea

Failures come in two kinds.

• Transient: a timeout, a deadlock, a brief outage. The same message will likely succeed on the next try.
• Permanent (poison): malformed JSON, a reference to a deleted user, a bug triggered by one input. It fails every time.

The system needs to retry the first kind and stop retrying the second. A dead-letter queue (DLQ) is the place where messages go when retrying has stopped.

Walking through backoff

Retrying immediately against a struggling database makes things worse. Instead, wait longer after each failure. A common rule is exponential backoff: double the wait each time.

def retry_delay(attempt, cap=900):
    """Seconds to wait after the given failed attempt: 2, 4, 8, ... capped at 15 minutes."""
    return min(cap, 2 ** attempt)


for attempt in range(1, 12):
    print(f"failed attempt {attempt:2d} -> wait {retry_delay(attempt):3d} s")

Output:

failed attempt  1 -> wait   2 s
failed attempt  2 -> wait   4 s
failed attempt  3 -> wait   8 s
...
failed attempt  9 -> wait 512 s
failed attempt 10 -> wait 900 s
failed attempt 11 -> wait 900 s

The cap stops the wait from growing past 15 minutes. In production, add jitter: a random amount added to each wait. Without it, a thousand messages that failed together all retry together, and the dependency is hit by a new spike.

The dead-letter queue

After a fixed number of attempts, the broker moves the message to a separate queue, the DLQ. The main queue keeps flowing. SQS calls the limit maxReceiveCount in its redrive policy. Typical values are 3 to 10.

Then three rules keep the DLQ useful:

• Alarm on DLQ depth. A DLQ nobody watches is a silent data-loss bin. Any message in it should page someone or open a ticket.
• Inspect before you redrive. Find out why the messages failed before moving them back.
• Redrive only idempotent work. Some messages may have partly succeeded before they failed. Replaying them is safe only if the handler is idempotent.

Worked case

A payment provider starts returning HTTP 503 for 10 minutes.

1. Each failing message retries after 2, 4, 8, 16 and 32 seconds, then more slowly.
2. The provider recovers. Most messages succeed on a retry and nothing reaches the DLQ.
3. One message has a currency code the provider doesn't support. It fails the same way every time, so after the limit it moves to the DLQ.
4. The alarm fires. An engineer fixes the currency mapping, then redrives that one message.

Only the poison message ended up in the DLQ. The rest of the traffic never stopped.

Common mistakes

• No backoff. Instant retries turn a brief outage into a longer one.
• No DLQ, or a DLQ with no alarm. Failures become invisible.
• Retrying forever. One poison message can block a whole ordered queue.
• Redriving without a fix. The messages go straight back to the DLQ.

Key takeaways

• Retry transient failures with exponential backoff and jitter.
• Stop after a fixed number of attempts, and move the message to a dead-letter queue.
• Alarm on the DLQ, inspect messages, then redrive only after a fix and only if the handler is idempotent.

Further learning

Next, see how many workers you need in backpressure and autoscaling.