Message ordering

A customer places an order, then cancels it. The messages are OrderPlaced then OrderCancelled. If the cancel is processed first, the system might ship an order that was already cancelled. Ordering is the promise that messages are processed in the order they were sent.

The big idea

By default, most queues do not promise order. Messages are spread across many servers and handled by many workers at once, so one message can finish before an earlier one.

So ordering is a trade-off. You pick how much order you need, and pay for it in throughput.

Three ways to get the order you need

1. Global FIFO. Every message goes through one lane, one at a time. Simple to reason about, but throughput equals one worker's speed. Use it only for low-volume streams.

2. Per-key ordering. Give each message a key, such as the customer ID. Messages with the same key are processed in order. Messages with different keys run in parallel. This is the useful compromise. SQS FIFO calls the key a message group ID. Pub/Sub calls it an ordering key. Kafka uses a partition key.

3. Order-insensitive consumers. Put a version number or timestamp on each update. The consumer ignores any update older than the one it already applied. Then order no longer matters, and you can use a cheap, unordered queue.

Walking through out-of-order retries

Retries break order even when the queue is ordered. Here is what happens to two updates for the same order.

Send:       update 2 (set status = packed)
            update 3 (set status = shipped)

Attempt:    update 2 fails (database timeout)
            update 3 succeeds   -> status = shipped
            update 2 retried    -> status = packed   <- wrong, shipped is newer

Final state: packed. The order really shipped.

Versioning fixes this. With version on each update, the consumer applies update 2 only if the stored version is below 2. After update 3, the stored version is 3, so the late update 2 is ignored.

stored version = 3 (shipped)
incoming update 2 (version 2) -> 2 < 3, ignore
final state: shipped  <- correct

The cost of per-key ordering

Per-key ordering has its own failure mode: head-of-line blocking. If one message for customer-42 keeps failing, every later message for that customer waits behind it. Other customers are unaffected, but that customer's queue is stuck until the bad message is removed or fixed.

Common mistakes

• Turning on FIFO by default. It caps throughput. Ask first what goes wrong if two messages for the same entity arrive out of order.
• Using a timestamp with clock drift. Two servers can disagree about the time. Use a version number from the database when you can.
• Forgetting retries. Ordering is lost on retry, even with a FIFO queue, unless the consumer checks versions.

Key takeaways

• Queues don't guarantee order by default.
• Strict global order costs throughput. Per-key order is the usual compromise.
• Retries can reorder even ordered queues, so use version numbers to make the consumer order-insensitive.

Further learning

Next, see how failing messages are handled in retries and dead-letter queues.