Message queues

When a user uploads a video, they shouldn't wait while it is transcoded into eight resolutions. When an order is placed, checkout shouldn't fail because the email service has a bad minute. The tool that handles both is a message queue: a buffer that holds work until a worker is ready to process it.

Why queues exist

A queue gives you four things.

• Decoupling. The producer only needs the queue to be up, not the consumer. Checkout publishes OrderPlaced and returns. Email, analytics and fulfilment pick it up whenever they can.
• Buffering bursts. Traffic is spiky, but databases and rate-limited APIs have fixed capacity. The queue absorbs the spike.
• Moving slow work off the request path. Thumbnails, emails and webhooks don't block the user.
• Automatic retries. A failed message stays in the queue and is delivered again.

The cost is that the system becomes asynchronous. You need to track status, watch queue depth and tolerate duplicates. The rest of this section explains those pieces one at a time.

A burst, worked out

Consumers handle 1,500 messages per second. Normal traffic is 1,000 per second. A flash sale sends 5,000 per second for 60 seconds.

During the burst (60 s):
  arrived    = 5,000 × 60 = 300,000
  processed  = 1,500 × 60 =  90,000
  backlog    = 300,000 − 90,000 = 210,000 messages

After the burst (traffic back to 1,000/s):
  spare capacity = 1,500 − 1,000 = 500 messages/s
  drain time     = 210,000 / 500 = 420 s = 7 minutes

Without a queue, you'd reject about 70% of orders during that minute. With a queue, every order is processed, the last ones about 7 minutes late. Whether 7 minutes is acceptable is a product decision, and a queue lets you have that conversation.

Concepts in this section

Read them in order. Each one builds on the one before.

For log-based systems like Kafka, see event streaming.

Common mistakes

• Dual writes. The service writes to its database, then publishes to the queue. If it crashes in between, the order exists but nobody hears about it. Use the transactional outbox: write the event in the same database transaction, and have a relay publish it. See distributed transactions.
• Treating the queue as a database. Queues are poor at "show all pending jobs for customer 42". Keep job state in a real table and use the queue for delivery.
• One shared queue for every job type. A slow job type delays everything else in the queue. Separate queues by priority or job class.

Key takeaways

• A queue decouples producers and consumers, absorbs bursts and makes retries automatic, at the cost of asynchrony.
• Most of the hard questions are about delivery: what happens on a crash, a duplicate, a failure, or an out-of-order message.
• Read the concept pages in the table above, starting with async work.

Further learning

Gaurav Sen's short video is a good intuition-builder for why queues exist and where they fit in a backend.