Sizing partitions and avoiding hot keys

Partition count is a capacity decision made early. Too few partitions cap your parallelism. Too many cost memory, file handles and end-to-end latency. And you can add partitions to a topic but never remove them. Size from numbers, with headroom.

Worked sizing example

Take the event pipeline for a ride-sharing app. Requirements: peak ingest of 200,000 events a second, 1 KB each, 7 days of retention, replication factor 3. Consumers enrich each event and write to a database. Benchmarks show one consumer handles 5 MB/s per partition.

Ingest throughput:
  200,000 events/s × 1 KB = 200 MB/s

Partitions for consumers:
  200 MB/s ÷ 5 MB/s per partition = 40 partitions

Partitions for producers (assume ~20 MB/s per partition is comfortable):
  200 MB/s ÷ 20 MB/s = 10 partitions

Take the larger number (40) and add headroom for 2 to 3 times growth:
  ≈ 100 to 120 partitions

Storage:
  200 MB/s × 86,400 s/day = 17.28 TB/day
  × 7 days                = 121 TB
  × 3 replicas            ≈ 363 TB on broker disks
  (with zstd at about 3 times compression ≈ 121 TB)

Network out of the leaders:
  replication: 200 MB/s × 2 followers = 400 MB/s
  each consumer group reads 200 MB/s; 3 groups = 600 MB/s

The arithmetic says the real constraint is storage. That is the case for tiered storage, or for a short local retention plus an archive in S3. It also gives you the network budget per broker, which often limits a cluster before CPU does.

Why partition count is hard to change

Adding partitions changes hash(key) mod N for most keys. A key that used to land in partition 7 may now land in partition 31. Its records are then split across two partitions, and their order is broken across the change. Compacted topics can end up with the same key in two partitions.

So pick a count with growth headroom on day one, but don't overshoot. A few dozen to a few hundred partitions per topic covers most needs.

Hot partitions

Keying assumes keys are roughly even. They often aren't. One celebrity account, one large merchant or one device stuck in a loop sends all its events to one partition. That consumer falls behind while the others sit idle.

Three options:

• Choose a finer key, if ordering allows. Use order_id instead of merchant_id.
• Salt the hot key. Spread merchant-9 across merchant-9#0 through merchant-9#7, which uses 8 partitions. Order then holds only within each salt bucket.
• Detect and route. Keep a small list of known hot keys and give them special handling.

The same skew problem appears in sharding, and the fixes are the same.

Common mistakes

• Sizing for average throughput. Size for the peak, then add headroom.
• Forgetting replication and retention multipliers. Storage is often several times the raw figure.
• Salting keys without accepting the cost. Ordering across the salt buckets is gone.
• Overshooting partition counts. Each partition costs memory, file handles and leader-election time.

Check yourself

1. Why is adding partitions risky for an ordered topic? It changes the hash modulo, so a key's records can move to a different partition, and their order breaks across the change.

2. One merchant produces 40% of all events. What do you try first? A finer key, if ordering allows, such as the order ID instead of the merchant ID.

Key takeaways

• Size partitions from peak throughput, consumer speed and storage, and state every multiplier.
• Storage and network are often the binding constraints, not CPU.
• Adding partitions remaps keys, so plan the count before the topic is busy.
• Hot keys need finer keys, salting, or special routing, and each option costs ordering somewhere.

Further learning

How to choose the number of topics and partitions in a Kafka cluster (Confluent)Confluent is building the foundational platform for data in motion so any organization can innovate and win in a digital-first world.Confluent