Pitfalls of event-driven systems

Event-driven systems fail in characteristic ways. Each failure has a known fix. Decide on the fix before you publish your first event, because retrofitting is painful, especially for personal data.

Debugging and observability

A synchronous call leaves a stack trace. An event chain has five consumers in four services, with a broker in the middle. Put a correlation ID and trace context, through OpenTelemetry, in every event's headers, and log it everywhere. Without that, an event-driven system is close to undebuggable.

Ordering

Kafka guarantees order only within a partition. If OrderPlaced and OrderCancelled for the same order land on different partitions, a consumer can see the cancellation first. Partition by the entity ID, such as the order ID, so each entity's events stay in order. Accept that there is no order across entities.

Consumers should also tolerate gaps. Carry a version number in each event, and ignore updates older than the version you already hold. See event streaming.

Duplicates

Delivery is at-least-once, so every consumer must be idempotent. Deduplicate by event ID, use upserts, or check versions. See idempotency.

Event soup

When everything is an event and anyone can subscribe to anything, the system becomes event soup. Nobody knows which services depend on which events. Events are published that nobody consumes. Changing a shape breaks consumers you have never met.

The countermeasures:

• A schema registry with compatibility rules. Confluent Schema Registry with Avro or Protobuf rejects incompatible changes.
• Documented ownership. Every topic has an owning team.
• An event catalog. A list of what exists, who publishes it and who reads it.
• Versioned public events. Treat them as an API, with the same discipline you apply to REST.

Deleting data from immutable logs

Privacy rules such as GDPR's right to erasure collide with "events are stored forever". Three options:

• Crypto-shredding. Encrypt each user's personal data inside events with a per-user key, stored separately. To "delete" the user, destroy the key. The ciphertext stays, but it can no longer be read.
• Keep personal data out of events. Store a reference such as customerId in the event, and keep the personal details in a store that you can delete from.
• Compacted topics with tombstones. For a Kafka topic that holds the current state per key, a null-value tombstone removes that key's earlier records during compaction.

Decide on one of these before the first event with an email address in it is published.

Common mistakes

• Partitioning by something other than the entity. Ordering then breaks for the entity you care about.
• No correlation ID. Debugging becomes guesswork.
• Personal data in events with no deletion plan. The first erasure request becomes a crisis.

Check yourself

1. Two events for the same order land on different partitions. What can go wrong? A consumer can see the cancellation before the placement. Partition by order ID so that each order's events stay in order.

2. What is crypto-shredding? Encrypting each user's personal data with a per-user key. Destroying the key makes the stored ciphertext permanently unreadable.

Key takeaways

• Carry a correlation ID through every event to make the system debuggable.
• Partition by entity to keep that entity's events in order, and version events to ignore stale updates.
• Assume duplicates, and make every consumer idempotent.
• Govern schemas and ownership so event soup doesn't form.
• Choose a personal-data deletion strategy before the first event is published.

Further learning

Next, see how these ideas fit together in a complete design in the order system worked example.