Gateways and connection registry

A normal web tier is stateless. Any request can go to any server. A real-time tier is stateful, because an open connection lives on exactly one machine. That changes the design. The usual answer is to split the tier into connection gateways, which hold sockets, and application services, which hold no connections.

The big idea

A gateway does as little as possible. It authenticates the connection, keeps the socket alive, forwards inbound messages to backend services, and writes outbound messages to the right sockets. Application services handle the business logic, such as "store this chat message, then deliver it to the channel". They never hold a connection.

The two tiers are linked by a pub/sub backbone, covered in fan-out with pub/sub. A connection registry tells the system which gateway holds each user.

Gateways are kept thin for a practical reason. Every gateway deploy disconnects its users. Business logic changes every day, but gateway code can stay the same for weeks.

Walking through it

The registry. To deliver a message to user 42, something must know which gateway holds user 42's connection. That is the registry, a fast store such as Redis that maps user_id to {gateway_id, connection_id}. One user can have several entries, for a phone, a laptop and a browser tab. Each entry has a TTL, and the gateway refreshes it about every 30 seconds:

gateway-17 refreshes  user:42 → gateway-17  (TTL 30 s)  every 25 s
gateway-17 crashes     → no refresh → entry expires after 30 s

If a gateway dies, its entries expire on their own. They do not keep pointing at a dead machine.

Sticky routing. The load balancer keeps each connection on one gateway. WebSockets are one TCP connection, so this is natural. Fallbacks that reconnect often, such as long polling, need sticky routing too, usually by hashing a user or session ID. Otherwise they hop between gateways and lose buffered messages. See load balancing.

How many connections per machine? An idle WebSocket costs a file descriptor, some kernel buffers and a little application memory. Very little CPU. Reported numbers are high. WhatsApp reported over 2 million TCP connections on one server in 2012, and the Phoenix team benchmarked 2 million WebSocket connections on one large server in 2015.

Memory is usually the limit. Here is a rough estimate:

Per connection:
  kernel socket buffers (tuned down)   ≈  8 KB
  TLS session state                    ≈ 10–40 KB, depends on the library
  application state                    ≈  2–10 KB
  total                                ≈ 20–60 KB

1,000,000 connections × 40 KB ≈ 40 GB of RAM

Most teams run 50,000 to 200,000 connections per gateway, even when a box could hold more. The reason is blast radius. If a gateway with a million connections dies, a million clients reconnect at once.

Common mistakes

• Putting business logic in gateways. Every change then disconnects users.
• A registry without TTLs. Dead gateways stay in the registry forever.
• One giant gateway. A single failure disconnects everyone on it.
• Ignoring the file descriptor limit. Raise it, or connections fail long before memory runs out.

Check yourself

1. Why does the registry need TTLs?

   Answer: if a gateway dies, its entries expire instead of pointing at a dead machine forever.

2. Estimate memory for 1 million connections at 40 KB each.

   Answer: about 40 GB (1,000,000 × 40 KB).

Key takeaways

• Real-time servers split into thin stateful gateways and stateless application services.
• A TTL-based registry maps each user to the gateway that holds their connection.
• Memory per connection sets how many connections one machine can hold.
• Keep gateway fleets moderate in size, to limit the blast radius of one failure.

Further learning

Slack's engineering account of gateway, channel and presence servers is the best production example of this split:

Real-time Messaging (Slack Engineering)Did you know that ground stations transmit signals to satellites 22,236 miles above the equator in geostationary orbits, and that those signals are then beamed down to the entire North American subcontinent? Satellite…slack.engineering

Next, see how one message reaches every gateway that needs it, in fan-out with pub/sub.