Guide

Architecture patterns

Why duplicates happen

Retries at every layer and at-least-once queues mean the same request can arrive twice. Learn where duplicates come from, and why the receiver must tolerate them.

170 of 285 · 2 min read

Duplicate requests feel like a rare edge case until you count them. A request can be repeated by your own code, by a client, by a load balancer or by a queue. Once you see every source, you stop asking whether duplicates will happen and start asking how the receiver will handle them.

Concept

A duplicate is a second copy of a request that the receiver has already seen, or may have already processed. It arrives because the sender could not tell whether the first copy succeeded.

Where duplicates come from

  • Your own retries. A timeout is an unknown outcome. The first attempt may have succeeded, so retrying can repeat it. (See partial failure.)
  • Client retries. A user double-clicks "Pay". A mobile app resends after a flaky connection drops the reply.
  • Infrastructure retries. A load balancer or service mesh retries a failed request on another backend.
  • Queue redelivery. Most queues and streams give at-least-once delivery. A consumer that crashes after processing a message, but before acknowledging it, sees that message again.

The last source is structural. Queues that never redeliver lose data on every crash. That is at-most-once delivery. Exactly-once delivery is impossible in general, as delivery semantics explains.

Walking through the scale

Suppose your service handles 10,000 requests per second, and 0.01% of calls time out.

10,000 requests/s × 0.0001 = 1 timeout per second
1 timeout/s × 86,400 s     = 86,400 unknown outcomes per day

Duplicates are not a rare event at that scale. They happen constantly, and each one is a chance to double-charge, double-ship or double-email.

The only robust strategy

You cannot stop the network, the client or the broker from repeating a request. You can make the receiver tolerate repeats. The rest of this topic covers how: first by designing operations that are safe to repeat, then by adding keys and deduplication where that isn't possible.

Common mistakes

  • Assuming the broker or network prevents duplicates. Neither does, in general.
  • Retrying at several layers at once. Each layer multiplies the others' retries. See retry budgets.
  • Treating a timeout as a failure. It is an unknown outcome, and it may have succeeded.

Check yourself

1. A consumer processes an order, then crashes before acknowledging the message. What happens next? The broker redelivers the message, so the work can run a second time.

2. Why isn't "never redeliver" a fix for duplicates? It turns every crash into a lost message. You trade duplicates for data loss.

Key takeaways

  • Duplicates come from your retries, client retries, infrastructure retries and queue redelivery.
  • At-least-once delivery means duplicates are normal, not rare.
  • Make the receiver tolerate repeats, rather than trying to stop them everywhere.

Further learning

Next, see which operations are safe to repeat by design in naturally idempotent operations.

Delivery semantics

The three promises a messaging system can make (at-most-once, at-least-once, exactly-once) and why the real choice is when the consumer says "done".

What goes wrong in distributed systems

Partial failure, unreliable networks, paused processes and timeouts, and the fencing tokens and failure detectors that let you build correct systems on top of them.

Idempotency and retries

Making operations safe to retry, with idempotency keys, deduplicating consumers and jittered backoff, so at-least-once delivery never means a double charge.