Retries and provider failures

Providers fail in several ways. A request might time out, a device token might be dead, a provider might be down, or a template might be broken. Each failure needs a different response. Retry transient errors with backoff and jitter, stop retrying permanent ones, and send anything that keeps failing to a dead-letter queue where an operator can see it.

The big idea

Sort each failure into a class before deciding what to do:

Infinite retry is the common failure. A template error retried forever wastes capacity and never succeeds.

Walking through it

Backoff spreads retries out in time. The usual pattern is exponential backoff with full jitter: each wait is a random amount up to a ceiling that doubles with every attempt. Jitter stops many failed sends from retrying at the same instant.

import random

def backoff_seconds(attempt, base=1.0, cap=60.0):
    # Full jitter: wait a random amount up to an exponentially growing ceiling
    ceiling = min(cap, base * 2 ** attempt)
    return random.uniform(0, ceiling)

random.seed(7)  # fixed seed, so the output is repeatable
for attempt in range(1, 4):
    print(attempt, round(backoff_seconds(attempt), 2))

Output:

1 0.65
2 0.6
3 5.21

The ceiling doubles each time, so attempt 3 can wait up to 8 seconds. Attempt 2 happened to draw a short wait. That randomness is the point. The cap keeps the wait from growing without bound.

Record every attempt. The log entry needs the notification ID, channel, attempt number, provider ID, status, latency and error code. This log powers support tools, and it answers "did the SMS actually leave our system?"

Device tokens need hygiene too. When APNs or FCM says a token is Unregistered or InvalidToken, delete it or mark it immediately. Bind tokens to the user and device, and send to every token a user has, unless the template is device-specific.

Common mistakes

• Retrying a template error. The same bad template fails the same way every time. Send it to the DLQ.
• Retrying without jitter. Clients that failed together retry together, and they overload the provider again.
• Dropping failures silently. Failed messages must land in a DLQ that someone watches.
• Ignoring DLQ depth. A growing DLQ is an alert, not a backlog to ignore.

Check yourself

1. Why does a template error belong in the DLQ and not in the retry loop?

   Retrying the same bad template fails the same way. An operator must fix the template, and the DLQ holds the messages until then.

2. Why add jitter to backoff?

   Without jitter, failed sends retry at the same moment and hit the provider together again.

Key takeaways

• Classify each failure, and respond to each class in its own way.
• Use exponential backoff with jitter and a capped number of attempts.
• Dead-letter queues hold what keeps failing, and an alert tells someone to look.
• Remove dead device tokens immediately, so you stop sending to them.

Further learning

Next: priority lanes and broadcasts, which covers how urgent messages avoid the backlog.