Change data capture

Most systems already have a database that is the source of truth. The question is how to get reliable events out of it. The naive approach is to write to the database and then publish to a broker. That is a dual write, and a crash between the two loses the event. Publishing first is worse, because the event may announce a change that never commits.

The big idea

Every committed change in Postgres goes into the write-ahead log (WAL). In MySQL, it goes into the binlog. A CDC tool such as Debezium connects as a replication client, reads committed changes in commit order, and publishes them as events to Kafka.

Because CDC reads only the committed log, it never publishes a rolled-back change, and it never misses a committed one.

Walking through a change event

An order moves from pending to paid. Debezium publishes something like this:

{
  "op": "u",
  "source": { "table": "orders", "lsn": 23891004, "ts_ms": 1767225600000 },
  "before": { "id": 8812, "status": "pending", "total_cents": 4900 },
  "after":  { "id": 8812, "status": "paid",    "total_cents": 4900 }
}

The event says what changed, in which table, at which log position. Consumers can compare before and after to see the difference.

Why an outbox usually sits in front

Raw row-change events expose your internal schema to every consumer. Rename a column, and every downstream team breaks. Two refinements fix this.

• The transactional outbox. Write a purpose-designed event row into an outbox table, in the same transaction as the business change. Have CDC publish only that table. Consumers get stable domain events, and you keep the atomicity of CDC. See the transactional outbox.
• A translation layer. A small stream processor converts raw table changes into domain events before other teams see them.

Where CDC is used

CDC is the standard tool for several jobs. It migrates data between systems, feeds search indexes and data warehouses, and invalidates caches. The consumers on the other end are covered in batch and stream processing.

Common mistakes

• Publishing raw table rows to every team. You couple every consumer to your table design.
• Ignoring schema changes. A renamed column changes every event that carries it. Plan the change with the consumers.
• Writing the event outside the change's transaction. That brings the dual write back in.

Check yourself

1. Why does CDC avoid the dual-write problem? It reads the database's own log of committed changes, so there is no second write to fail.

2. Why put an outbox table in front of CDC? So consumers see stable domain events, not the internal layout of your tables.

Key takeaways

• CDC publishes committed database changes by reading the write-ahead log or binlog.
• It never publishes a rolled-back change, and it never misses a committed one.
• Raw row events couple consumers to your schema, so put an outbox in front.

Further learning

The Debezium documentation covers connectors, the outbox event router and the operational details.

Next, see what happens when the events themselves are the source of truth, in event sourcing.