Traffic and messaging
Event streaming and Kafka
How a durable log lets many consumers read the same events at their own pace, and how Kafka builds that log from partitions, replicas and consumer groups.
A message queue hands each message to a consumer and then forgets it. That is good for spreading work, but it throws away history. What if a new service wants every order from the last week? What if a bug corrupted the search index, and you want to rebuild it from the original events? What if five teams each want to read the same stream at their own pace?
Event streaming keeps the history. Its core structure is the log: an append-only, ordered sequence that many readers can consume independently and replay. Apache Kafka, created at LinkedIn around 2010, made the log the backbone of modern data infrastructure.
Reading order
| Step | Concept | What it answers |
|---|---|---|
| 1 | The log as a data structure | What is a log, and how does it differ from a queue? |
| 2 | Kafka architecture and replication | How are topics split, copied and kept durable? |
| 3 | Producers, keys and batching | Which partition does a record go to, and how is it sent? |
| 4 | Consumer groups, offsets and lag | How do consumers share work and track their place? |
| 5 | Exactly-once inside Kafka and its limits | What does "exactly-once" cover, and where does it stop? |
| 6 | Sizing partitions and avoiding hot keys | How many partitions do you need, and what breaks at scale? |
Read the first two before the table's later steps. Each later step assumes them.
Alternatives to Kafka
| System | Design | Notes |
|---|---|---|
| Apache Kafka | Brokers with local disks, replicated partitions, KRaft metadata | The de facto standard, with a large ecosystem. Managed versions exist from several vendors. |
| Amazon Kinesis Data Streams | Fully managed shards | Each shard takes about 1 MB/s or 1,000 records/s in, and 2 MB/s out. On-demand mode scales shards for you. AWS-only, with less flexible retention and tooling. |
| Apache Pulsar | Stateless brokers, storage in Apache BookKeeper | Separating compute from storage makes rebalancing easier. Built-in multi-tenancy and geo-replication, with more moving parts. |
| Redpanda | Kafka API, written in C++, Raft per partition | Works with Kafka clients, often with lower tail latency, and simpler to run. |
| Object-storage-backed logs (WarpStream, Confluent Freight, AutoMQ, Bufstream) | Stateless agents, data written straight to S3 | Avoids broker disks and most cross-zone replication traffic, so much cheaper at scale. End-to-end latency is hundreds of milliseconds, not single digits. |
In the cloud, the largest Kafka bill is often cross-zone network traffic for replication and consumer reads, not compute. Object-storage designs remove most of that cost, at the price of latency. For logging, analytics and change data capture, where half a second is fine, they are worth evaluating. For latency-sensitive event processing, classic Kafka or Redpanda still wins.
Mistakes that cause the most damage
- Using Kafka as a job queue. For background jobs with per-task retries and delays, a queue or job library is simpler.
- Keying by a field with few values. A merchant ID that dominates traffic makes one partition hot. See sizing partitions.
- Leaving auto-commit on for important data. Offsets are committed whether or not processing finished.
- Treating exactly-once as covering emails and API calls. It stops at Kafka's edge.
- Adding partitions to an ordered topic without a plan. Existing keys can move, and their order breaks across the change.
Key takeaways
- A log is append-only, ordered and replayable. Readers track their own offsets, so adding a consumer costs the producer nothing.
- Kafka splits each topic into replicated partitions. Ordering holds within a partition only.
- The key chooses the partition, so it decides what stays in order.
- Consumer groups share partitions. Their parallelism is capped at the partition count.
- Exactly-once in Kafka covers Kafka-to-Kafka work. External side effects still need idempotent sinks.
Further learning
Apache Kafka documentationApache Kafkakafka.apache.orgStart with the log as a data structure and work down the table.
Related
Message queues
A durable buffer between the code that creates work and the code that does it. Start here for the map of queue concepts.
Event-driven architecture, CQRS and event sourcing
Building systems around events, from change data capture to event sourcing and read models, and knowing when it simplifies a system versus creating a distributed mess.
Batch and stream processing
How to move and transform large data sets reliably. Batch jobs run over bounded data, streams run over events as they arrive, and the same ideas of time, state and idempotent writes apply to both.