Idempotency for notifications

Producers retry. Queue workers retry. Providers retry. Without care, one request turns into several sends, and a user gets five one-time codes. An idempotency key lets the system recognize a repeat and skip it. The goal is at most one send per key within a time window, not exactly-once delivery across the whole system.

The big idea

The system stores each key with the notification it created, for a limited time. When a request arrives with a key it has already seen, it returns the stored result and does not send again. The window matters: transactional sends might keep keys for 24 to 72 hours, and some marketing sends keep them shorter.

You can't get true exactly-once across your system and a provider such as APNs or Twilio. Once a provider accepts a message, you can't take it back. A crash between the send and the log can produce a rare duplicate. The aim is exactly-once effects: at most one send per key, and rare duplicates only in that crash case.

Walking through it

An OTP login flow shows the check. The auth service sends the same request twice, because the first response was lost:

1st POST /notify  key=otp:user_42:challenge_9f3a
  key not seen -> store key -> enqueue SMS -> return notification_id n_801

2nd POST /notify  key=otp:user_42:challenge_9f3a   (retry after a timeout)
  key seen     -> return the stored notification_id n_801
  no second SMS, the user receives one code

The key has to be stable across retries. The auth service must reuse the same otp:user_42:challenge_9f3a value on every attempt, not generate a new one. If the producer has no natural key, derive one from the business facts, such as a hash of the user, the template, a dedupe window and the business ID.

The check happens before the provider call. The worker looks up the key, sends if it's new, and records the outcome. Providers often accept an idempotency header too, so pass the key along. Use both layers.

Common mistakes

• Generating a random key on every retry. Each retry then looks new, and the user gets several codes.
• Deduplicating on a message ID alone. Use a stable business key, such as the order plus the action, or a row with a unique constraint.
• Skipping the provider's header. It is a second layer of protection for the crash case.
• Forgetting the key's lifetime. Without a window, the store grows forever. With the window too short, a late retry becomes a duplicate.

Check yourself

1. A producer retries an OTP request with a new random key each time. What happens?

   Each retry looks new, so the user can receive several codes. The key must stay the same across retries.

2. Why does the time window for keys matter?

   Keys are kept for a set period, such as 24 to 72 hours for transactional sends. After that, a repeat counts as a new request.

Key takeaways

• An idempotency key turns a retry into a no-op, as long as the key is stable and still stored.
• The realistic goal is at most one send per key, not exactly-once delivery across providers.
• Use both your own check and the provider's idempotency header.
• Choose the key and its window deliberately, and expire keys on purpose.

Further learning

Next: rate limits and provider quotas, which covers how fast the pipeline may call each provider.