Deduplicating in message consumers

Consumers of queues and streams face the duplicate problem from the other side. A message can arrive twice, so the consumer has to recognize the second copy and do nothing with it. The safest place to do that check is in the same database transaction as the work itself.

Technique 1: a processed-message table

Record each message ID in the same transaction as the side effect:

BEGIN;
  INSERT INTO processed_messages (consumer, message_id) VALUES ('email-sender', 'evt_91c2');
  -- Fails with a unique violation if this message was already handled.
  -- The consumer then rolls back and acknowledges the duplicate.
  UPDATE orders SET status = 'paid' WHERE id = 42;
COMMIT;

If the consumer crashes after COMMIT but before acknowledging, the redelivered copy hits the unique constraint. The consumer acknowledges it and moves on. Because the dedup row and the state change commit together, there is no moment when one exists without the other.

Technique 2: a unique constraint on the business key

Often the message already carries a natural key, such as "create shipment for order 42". A unique index on shipments.order_id makes the second insert fail harmlessly. No extra table is needed.

Technique 3: the inbox pattern

Write each incoming message to an inbox table first, keyed by message ID. Acknowledge the broker, then process from the inbox. Duplicates are rejected at the insert.

This is the mirror image of the outbox, which writes outgoing events in the same transaction as the state change, so they can't be lost. See the transactional outbox.

When the side effect is outside your database

Some side effects can't share a transaction with your database, such as sending an email or calling a partner API. Three options:

• Pass an idempotency key derived from the message ID, such as email:evt_91c2, if the external system accepts one.
• Mark processed first. You risk losing the email if the send fails.
• Send first, then mark processed. You risk sending a duplicate email.

For a receipt email, a rare duplicate is the right trade-off. For a payment, use an idempotency key.

Effectively once

Processing that is at-least-once plus idempotent gives effectively-once effects. This follows the end-to-end argument from Saltzer, Reed and Clark's 1984 paper. A guarantee built into the middle of a system can't replace a check at the endpoint that changes state. Each place where a side effect happens needs its own deduplication, and the cheapest place to put it is usually the place that owns the state.

Common mistakes

• Recording the message ID separately from the work. A crash between the two either loses the work or repeats it.
• Pruning processed IDs too early. Keep each ID at least as long as the broker can redeliver that message.
• Assuming an external call is atomic with your transaction. It is not. Make it idempotent.

Check yourself

1. Why must the processed-ID insert and the side effect share a transaction? If they are separate, a crash between them either repeats the effect (effect done, ID not saved) or loses it (ID saved, effect not done).

2. When can you delete old processed IDs? Only after the broker can no longer redeliver that message.

Key takeaways

• Record processed IDs in the same transaction as the side effect.
• A unique constraint on a business key can do the same job with no extra table.
• The inbox pattern deduplicates incoming messages before processing them.
• Exactly-once effects come from at-least-once delivery plus idempotent processing.

Further learning

Kafka's exactly-once guarantee covers only what happens inside Kafka. See exactly-once in Kafka for where its boundary sits.

Next, see how to space out retries in retry policy and backoff with jitter.