Traffic and messaging
Queues vs pub/sub topics
Two ways to deliver a message. A queue gives each message to one worker. A topic gives a copy to every subscriber. Real systems use both.
Imagine an order is placed. Billing needs it, shipping needs it, and analytics wants it. Should one worker take the message, or should all three teams get it? The answer depends on the shape of the work, and that is the difference between a queue and a pub/sub topic.
The big idea
- Queue (point-to-point): each message goes to one consumer. Several workers compete for messages, so each job is done once.
- Topic (pub/sub): each message goes to every subscriber. Each subscriber gets its own copy and processes it independently.
Concept
Competing consumers means several workers pull from the same queue. Each message is handled by exactly one of them. This is how you spread a backlog across a pool of workers.
A mail sorting office is a useful picture. A queue is a single pile of parcels that several clerks take from. A topic is a photocopier: every department gets its own copy of the same memo.
Walking through a single order
Here is what happens when OrderPlaced is published to a topic that has three subscriptions.
- The topic makes three copies: one per subscription.
- Billing has two workers that share its copies. Each billing message is handled by one of them.
- If email fails, only the email copy is retried. Billing and shipping are unaffected.
Most real systems combine both. A topic fans out to several queues, and each queue has a pool of competing consumers. On AWS, that is SNS fanning out to several SQS queues. In Google Pub/Sub, each subscription acts like its own queue.
When to pick which
| Situation | Use | Why |
|---|---|---|
| Resize one uploaded image | Queue | Only one worker should do it |
| Tell billing, shipping and analytics about an order | Topic | Each team needs its own copy |
| Process 10,000 emails with 20 workers | Queue | Work is split across workers |
| Any new team might want the event later | Topic | New subscriptions don't change the producer |
Trade-off
A topic decouples producers from consumers more strongly, because adding a subscriber needs no change to the producer. The cost is that every subscriber has its own backlog, retries and failures to watch.
Common mistakes
- Using a queue when several services need the event. Each extra service then needs a change in the producer or a hacky second queue. Use a topic.
- Using a topic for work that should happen once. If all subscribers do the same job, you'll charge the card three times. Put the work on one queue.
- Assuming a topic keeps history. Many topics only deliver messages to subscriptions that exist when the message is published. Check the retention rules for your product.
Key takeaways
- A queue gives each message to one consumer. A topic gives a copy to each subscriber.
- Competing consumers split the work on one queue.
- Combine the two: a topic fans out to queues, and each queue has its own workers.
Further learning
Next, learn what "delivered" really means in delivery semantics.
Related
Message queues
A durable buffer between the code that creates work and the code that does it. Start here for the map of queue concepts.
Message ordering
Why messages can arrive out of order, what it costs to guarantee order, and how to design consumers that don't need it.
Event streaming and Kafka
How a durable log lets many consumers read the same events at their own pace, and how Kafka builds that log from partitions, replicas and consumer groups.