Idempotency and retries

Your service calls a payment provider to charge a customer $49. The call times out after 10 seconds. Did the charge happen? You cannot know. The request may have been lost on the way out. The provider may have charged the card and crashed before replying. The reply may be sitting in a congested buffer. Retry, and you risk charging twice. Don't retry, and you may ship an order nobody paid for.

This is the central problem of distributed systems, and the answer is a two-part recipe. Retry, because failures are common and usually transient. And make the operation idempotent, so doing it twice has the same effect as doing it once. The topic covers both halves: where duplicates come from, which operations are safe to repeat, how to make unsafe ones safe, how consumers deduplicate, and how to retry without causing an outage.

Reading order

Read the first two before the hub's table. Steps 3 to 6 build on them.

Exactly-once is an end-to-end property

Vendors advertise "exactly-once", and it is worth being precise. Kafka's exactly-once semantics give you exactly-once inside Kafka, from one topic to another. The moment a consumer writes to Postgres, charges a card or sends an email, Kafka's guarantee stops at the boundary. (See exactly-once in Kafka.)

What you can build is effectively-once processing:

at-least-once delivery  +  idempotent processing  =  effectively-once effects

The broker guarantees each message arrives at least once. Your dedup table, unique constraints or downstream idempotency keys make the extra arrivals harmless. Each hop with a side effect needs its own protection.

Mistakes that cause the most damage

• A new key for every attempt. It defeats the whole mechanism. Generate the key once per logical operation.
• Assuming the broker prevents duplicates. Your consumer must tolerate them.
• Retrying errors that can't change. A 400, 404 or 422 will fail the same way every time.
• Retrying at every layer. Each layer multiplies the calls to the layer below.
• Assuming a database transaction makes an external call atomic. It can't roll back an HTTP call to a card network. Each external call has to be idempotent on its own.

Key takeaways

• A timeout is an unknown outcome. Retries are inevitable, so duplicates are too. Design the receiver to tolerate them.
• Prefer designs that are naturally idempotent: absolute sets, upserts on natural keys, conditional writes and ledgers keyed by transaction ID.
• Use idempotency keys when the operation can't be redesigned, and pass a derived key to every downstream system that has side effects.
• Retry only retryable errors, with exponential backoff and full jitter, at one layer, inside a retry budget.

In practice

• Public APIs that create things. Any POST that creates an order, payment, transfer, booking or message should accept an idempotency key. Adding one later breaks clients that already retry.
• Webhooks. Providers deliver webhooks at least once. Store the event ID and skip duplicates.
• Background jobs. Frameworks such as Sidekiq, Celery and SQS workers retry failed jobs. Make reruns harmless, with state checks, upserts and deterministic IDs.
• Mobile clients. Flaky networks mean the app often doesn't know whether a request landed. Generate the key on the device and keep it across restarts.
• Interviews. When a design moves money or creates records, say what happens on a retry. Name the key, where it is stored, how concurrent duplicates are handled and how long keys live.

Further learning

Start with why duplicates happen.