Kafka architecture and replication

Kafka is a set of servers called brokers that store logs. A topic is a named stream, such as orders. Each topic is split into partitions, and each partition is an ordered log that is copied to several brokers. Those choices decide what stays in order, what survives a broker failure, and how much parallelism you get.

Topics, partitions and segments

On disk, a partition is a directory of segment files, 1 GB each by default, with indexes from offset to file position. Writes are sequential appends. Reads are mostly sequential too, and are often served from the operating system's page cache. Kafka also uses zero-copy transfer (sendfile) to move bytes from the page cache straight to the network socket.

Sequential I/O, batching and zero-copy are why a modest cluster can sustain hundreds of MB/s, and a larger one several GB/s.

Leaders, followers and replicas

Each partition is copied to several brokers. The number of copies is the replication factor, usually 3. One copy is the leader. It handles writes and, by default, reads. The others are followers. They continuously fetch new records from the leader.

The in-sync replica set (ISR) is the set of copies that are caught up with the leader, within replica.lag.time.max.ms, 30 seconds by default. Two settings make a write durable:

• The producer sets acks=all. The leader confirms a write only after every replica in the ISR has it.
• The topic sets min.insync.replicas=2. If the ISR shrinks below 2, the leader rejects writes instead of accepting data that lives on one machine.

Consumers read only up to the high watermark, the last offset every in-sync replica has. They never see a record that a leader failover could erase.

Walking through a broker failure

With replication factor 3, acks=all and min.insync.replicas=2:

3 copies of partition 4, on brokers 1, 2 and 3. Broker 1 leads.
Broker 3 dies:      ISR = {1, 2}   ≥ 2 → writes continue, nothing acknowledged is lost
Broker 2 also dies: ISR = {1}      < 2 → leader rejects writes until a broker returns

So you can lose one broker without losing acknowledged data or availability. Lose two, and the partition becomes read-only until a broker comes back. That is the consistency over availability choice, made explicit.

KRaft: no more ZooKeeper

Kafka once depended on a separate ZooKeeper cluster to store metadata, such as which brokers exist and which replica leads each partition. KRaft replaced it with a quorum of Kafka controllers that keep metadata in an internal log, replicated with Raft. See consensus for Raft. Kafka 4.0, released in March 2025, removed ZooKeeper support. You now run one system instead of two. Metadata changes propagate faster, and failover after a controller loss takes seconds.

Common mistakes

• Using acks=all without min.insync.replicas of 2 or more. If the ISR has only the leader, acks=all is satisfied by the leader alone.
• Running replication factor 1 in production. One broker failure then loses the partition's data.
• Expecting order across partitions. Only records in the same partition are ordered.

Check yourself

1. With replication factor 3, acks=all and min.insync.replicas=2, how many brokers can fail without losing acknowledged writes? One. A second failure leaves the ISR below 2, so the partition becomes read-only until a broker returns.

2. Why must consumers read only up to the high watermark? Records past it are not yet on every in-sync replica. If the leader failed, those records could vanish after a consumer had already processed them.

Key takeaways

• A topic is split into partitions. Each partition is an ordered log, copied across brokers.
• acks=all with min.insync.replicas=2 and replication factor 3 survives one broker failure without losing acknowledged writes.
• The high watermark marks the records that are safe for consumers to read.
• KRaft replaced ZooKeeper. Kafka 4.0 removed ZooKeeper entirely.

Further learning

Tim Berglund's lightboard video from Confluent is a clear 10-minute tour of topics, partitions, producers and consumers.

Next, see how producers choose a partition in producers, keys and batching.