Traffic and messaging
Choosing a queue product
How Amazon SQS, RabbitMQ, Google Cloud Pub/Sub and Redis Streams differ, and how to pick one based on the shape of your work rather than on brand names.
Every queue product does the same basic job: hold messages, deliver them to workers, and track acknowledgements. The differences are in who runs it, how it routes messages, and what it guarantees. Pick the product after you know the shape of your work.
The big idea
Ask three questions before you look at a product:
- Who operates it? A managed service removes the ops work. A self-hosted broker gives you control.
- How do messages route? One consumer per message, fan-out to many, or complex routing rules?
- Do you need replay? Can a new consumer read history, or is a message gone once it is acked?
Concept
Replay means reading messages again from a stored history, usually to rebuild state or start a new consumer. Queues usually delete acked messages. Logs such as Kafka keep them. See event streaming.
Comparing the four
| Amazon SQS | RabbitMQ | Google Cloud Pub/Sub | Redis Streams | |
|---|---|---|---|---|
| Model | Queue (pair with SNS for fan-out) | Broker with exchanges that route to queues | Topics with subscriptions | Append-only stream with consumer groups |
| Operations | Fully managed | You run it, or use a managed version | Fully managed, global | Part of your Redis deployment |
| Delivery | At-least-once; FIFO queues add a dedup window | At-least-once with manual acks | At-least-once; exactly-once on pull subscriptions is optional | At-least-once via pending entries and XACK |
| Ordering | Best-effort; per message group in FIFO | Per queue, one consumer | Per ordering key, opt-in | Per stream |
| Retention | 1 minute to 14 days | Until consumed | Up to 7 days for unacked messages | Until trimmed |
| Good fit | AWS workloads, minimal ops | Complex routing, low latency, on-prem | GCP workloads, global fan-out | Lightweight jobs when Redis is already running |
Product limits change over time, so check the current documentation before you design around a number.
Walking through a decision
Suppose you run on AWS, you have no ops team, and you need to send emails for each order.
- Operations: no ops team, so a managed service.
- Routing: one email per order, plus a few other teams need the event. You need fan-out, so SNS plus SQS.
- Replay: not needed. Once an email is sent, the message can go.
The answer is SNS to several SQS queues. Had you needed replay for a new analytics team, Kafka or Kinesis would be a better fit.
Large messages
Most queues limit message size. SQS messages are capped at 1 MiB. For larger payloads, store the file in object storage and put a pointer in the message. This pattern is called the claim check.
Tip
Don't pick a queue because a team you admire uses it. Write down your answers to the three questions above, and check which products meet them.
Common mistakes
- Choosing for features you won't use. Complex routing is expensive to operate if you have one queue.
- Assuming "FIFO" means exactly-once. It means ordering within a group, and a dedup window. It does not stop a consumer from processing a redelivered message twice.
- Using Redis as a durable queue without thinking about persistence. Check how your Redis is configured before you trust it with orders.
Key takeaways
- The queue products differ in operations, routing, ordering and retention.
- Choose by the three questions: who runs it, how it routes, and whether you need replay.
- For large payloads, store the data elsewhere and send a pointer.
Further learning
Next, see how multi-step processes survive crashes in durable workflows.
Related
Message queues
A durable buffer between the code that creates work and the code that does it. Start here for the map of queue concepts.
Queues vs pub/sub topics
Two ways to deliver a message. A queue gives each message to one worker. A topic gives a copy to every subscriber. Real systems use both.
Event streaming and Kafka
How a durable log lets many consumers read the same events at their own pace, and how Kafka builds that log from partitions, replicas and consumer groups.