Large channels and receipts

Fan-out is how one message reaches many recipients. For a small group, the server can loop over the members, find their sessions, and push to each. That breaks down in a large channel with thousands or millions of members. Read receipts have the same problem. Tracking every member's read state for every message creates a write storm.

The big idea

Think of a radio station and a phone tree. A phone tree calls every listener one by one, so it slows down as the list grows. A radio station broadcasts once, and anyone tuned in hears it. Large channels work like the radio station: publish once to a topic, and let interested gateways pick it up.

The analogy breaks when people join and leave. Gateways must subscribe when a user joins and unsubscribe when they leave. Forgetting either step causes stale deliveries.

Walking through it

Take a 50-person group. Alice sends one message. The service writes one row to the channel's bucket. The registry finds 40 online sessions spread across 12 gateways, so the service makes 12 publishes. The 10 offline members each get one collapsed push, such as "3 new messages in #design." Every client drops duplicates by messageId.

For a large channel, don't loop over members on every message. Gateways subscribe to the channel topics their connected users care about. History still pages from the store. Live traffic is a pub/sub layer on top.

Receipts. A per-message receipt in a 200-person group costs 200 writes per message. Use these optimizations instead.

• Watermarks. Store one value per user and channel: "I have read up to sequence S." One write covers many messages.
• Throttled updates. Send receipt changes every few seconds while the channel is open.
• No receipts in huge channels. Skipping them is a product decision, and a reasonable one.

Common mistakes

• Looping over all members on every message. At a million members, each message becomes a million lookups.
• Per-message read receipts in big groups. The write count grows with membership times messages.
• Forgetting to unsubscribe. A gateway that keeps a topic after the user leaves keeps delivering messages nobody should see.

Check yourself

1. In a 200-person group, how many writes does per-message receipts cost? 200 writes per message, one for each member's read state.
2. What does a watermark store? The highest sequence number a user has read in a channel, so one write covers every message up to it.

Key takeaways

• Small groups fan out per member. Large channels publish once to a topic.
• Gateways subscribe when users join and unsubscribe when they leave.
• Store read state as watermarks, not per-message receipts.
• Skip receipts in very large channels when the product allows it.

Further learning

Real-time Messaging (Slack Engineering) explains gateway and channel fan-out in a workplace chat product.

Next, return to the chat system overview to see how these pieces fit together.