Producers, keys and batching

A producer decides three things for every record: which partition it goes to, how it is batched, and how durable the write must be. The key decides the partition, and therefore the ordering scope. Batching decides throughput. Acks decide durability. Idempotence decides whether a retry creates a duplicate.

Partitioning by key

The default partitioner hashes the key with murmur2 and takes the result modulo the partition count. All records with the same key go to the same partition, so they stay in order.

Choose the key by asking "what must be ordered?". Use order_id for an order's lifecycle events, user_id for one user's activity, and account_id for ledger entries. Records with no key are spread across partitions. Modern clients use a sticky partitioner, which fills a batch for one partition before moving to the next. That makes batches larger than strict round-robin would.

Batching and compression

Producers don't send records one at a time. They collect records per partition into batches. Two settings govern this:

• batch.size is the most data to collect, 16 KB by default.
• linger.ms is how long to wait for a batch to fill. Kafka 4.0 raised the default from 0 to 5 ms.

Batches are compressed as a unit with lz4 or zstd. That works well on repetitive JSON, often making it 3 to 5 times smaller. Raising linger.ms to 10 to 20 ms and batch.size to 64 to 256 KB is the biggest throughput lever for most producers. The cost is a few milliseconds of latency.

Acks: how durable is a write?

Idempotent producers: no duplicates from retries

A producer that retries after a timeout can create a duplicate. The first attempt may have succeeded, but its acknowledgement was lost. The idempotent producer fixes this. Each producer gets an ID and numbers its records per partition. The broker discards any record whose number it has already written. Since Kafka 3.0, enable.idempotence=true and acks=all are defaults, so a modern producer gets durable, duplicate-free, in-order writes per partition by default.

A sensible high-throughput configuration:

acks=all
enable.idempotence=true
compression.type=zstd
linger.ms=20
batch.size=131072
delivery.timeout.ms=120000

Common mistakes

• Keying by a field with few values. A single large merchant can make one partition hot. See sizing partitions.
• Turning off idempotence to save overhead. You trade a small cost for duplicates on every retry.
• Using acks=1 for data you can't lose. A leader crash can erase the record before followers copy it.
• Expecting order across keys. Ordering is per partition, so records with different keys can interleave.

Check yourself

1. Why does keying events by order_id keep an order's events in sequence? All records with the same key hash to the same partition, and a partition is ordered.

2. Why does a retry create a duplicate without idempotence? The first attempt may have been written, but its acknowledgement was lost. The retry writes the record again.

Key takeaways

• The key chooses the partition, so it sets the ordering scope.
• Batching and compression are the main throughput levers, at a small latency cost.
• acks=all with enough in-sync replicas makes writes durable.
• Idempotent producers stop retries from creating duplicates within a partition.

Further learning

Next, see how consumers share partitions and track their place in consumer groups, offsets and lag.