The log as a data structure

A log is the simplest storage structure there is. Records are added at the end, never changed, and each gets a sequence number. Many databases use one internally, and event streaming makes it the thing you build on directly.

The big idea

A log has three rules:

• Records are appended to the end and never modified.
• Each record gets a sequential offset.
• Readers read forward from any offset, in order, and track their own position.

offset:  0        1        2        3        4        5
       [order1] [order2] [order3] [order4] [order5] [order6]  ← producers append here
                  ▲                          ▲
          analytics reads here       billing reads here

Two readers can sit at different offsets and never interfere. Analytics can be an hour behind billing, and the producer doesn't know either of them exists.

That gives three useful properties. Every reader that processes the same log from the same offset reaches the same state. Adding a tenth reader costs the producer nothing. And because records are kept, a reader can rewind and replay.

Streams versus queues

The last row is the real trade-off. A queue tracks each message, so one slow message doesn't hold up the rest. A log tracks only a position per partition. That is cheap, but you must handle a poison record yourself, usually by writing it to a separate dead-letter topic and moving on.

How long the log keeps data

A log keeps records according to the topic's cleanup policy.

• Delete. Records expire after a time, 7 days by default, or when a partition exceeds a size limit. This suits click streams and logs.
• Compact. The log keeps at least the latest record for each key and removes older ones in the background. A record with a null value, called a tombstone, deletes the key.

offset 0: user 7 → {"plan": "free"}
offset 1: user 9 → {"plan": "pro"}
offset 2: user 7 → {"plan": "pro"}
offset 3: user 9 → null   (tombstone: user 9 is deleted)

After compaction: offset 0 is removed because offset 2 replaces it for user 7.
A reader from offset 0 rebuilds the current state: user 7 is pro, and user 9 is gone.

A compacted topic is effectively a replicated, replayable key-value table. Long retention lets the log be the source of truth, and services rebuild their own databases and indexes by replaying it. Tiered storage moves older segments to object storage such as S3, which keeps long retention affordable.

Common mistakes

• Querying the stream directly. Build a read model in a database instead.
• Assuming long retention is free. Storage and replication cost grows with retention. Use tiering.
• Expecting per-message acknowledgement. A log moves a single position per partition, so a stuck record holds up its partition.

Check yourself

1. Why can adding a tenth reader cost the producer nothing? Each reader tracks its own offset, so readers don't interact with each other or with the producer.

2. A compacted topic has two records for user 7: offset 0 with plan free, and offset 2 with plan pro. What does a new reader rebuild? Only the latest value after compaction: user 7 on plan pro. Before compaction runs, the reader may still see both records, so it must process them in order.

Key takeaways

• A log is an append-only sequence where readers track their own offsets.
• Readers can replay history, and adding readers doesn't burden the writer.
• Queues delete on consumption and track each message. Logs keep records and track positions.
• Compaction keeps the latest value per key, and a tombstone deletes a key.

Further learning

Jay Kreps's essay on the log, expanded into a short book, is the foundation for why Kafka looks the way it does.

His Strange Loop talk on turning the database inside out is the classic case for building systems around a log.

Next, see how Kafka builds this log from partitions and replicas in Kafka architecture and replication.