Exactly-once inside Kafka and its limits

Kafka offers exactly-once semantics (EOS). Be precise about what they cover, because the phrase is easy to over-read. Inside Kafka, the guarantee is real. Once a write leaves Kafka, the guarantee ends.

What is covered

For a read-process-write app, the producer writes its output and commits the consumer's offset in one transaction. Consumers of the output topic with isolation.level=read_committed see records only once the transaction commits. Kafka Streams turns this on with processing.guarantee=exactly_once_v2.

Here is one order, handled by an app that totals orders:

1. read order 881 from topic orders (offset 40)
2. write its total to topic totals, inside a transaction
3. commit: the total record and offset 41 become visible together

If the app crashes before step 3, the transaction aborts.
The totals record never becomes visible, and offset 40 is read again.

No duplicate total ever appears to a read_committed reader.

Where the guarantee stops

Kafka transactions can't include anything outside Kafka. If your consumer sends an email, calls a payment API or writes to Postgres, the transaction doesn't cover it. A crash after the side effect and before the offset commit repeats the side effect.

For external sinks, use one of these:

• Idempotent writes. Upsert by a natural key or event ID, so a replay overwrites the same row.
• Store the offset with the data. Write the result and the consumed offset in one database transaction. On restart, read the offset from the database rather than from Kafka. Many sink connectors work this way.
• Idempotency keys for external APIs. See idempotency keys.

The summary is the same as in consumer deduplication. Exactly-once is at-least-once delivery plus de-duplication. Kafka does the de-duplication for you only for writes back into Kafka.

Common mistakes

• Believing EOS covers emails and API calls. It covers only what Kafka writes.
• Forgetting read_committed on downstream consumers. They can see output from transactions that later abort.
• Choosing EOS when an upsert would do. A natural-key upsert on the sink is often simpler and just as safe.

Check yourself

1. Your consumer sends a receipt email, then crashes before its offset commits. What happens on restart? The message is processed again, and the email is sent twice. The email is outside Kafka's transaction, so it isn't covered.

2. What does read_committed change for a downstream consumer? It hides records from transactions that were aborted or are still open, so the reader sees only committed output.

Key takeaways

• Kafka's exactly-once covers reads and writes inside Kafka, using transactions.
• Side effects outside Kafka, such as email, payments or database writes, are not covered.
• Make external sinks idempotent, or store offsets in the same database as the result.

Further learning

Martin Kleppmann's keynote asks whether Kafka is a database, and walks through what a log can and can't give you for consistency and transactions.

Next, see how to size a topic in sizing partitions and avoiding hot keys.