Connection lifecycle

Connections die quietly. A phone walks into a tunnel, a NAT router forgets the mapping, and neither side gets a reset. The lifecycle of a connection needs three things: a way to detect dead connections, a way to resume without losing messages, and a way to reconnect without overwhelming the servers.

The big idea

Both sides send heartbeats, small pings that prove the connection is alive. Every message gets a sequence number, so the client can say "resume after 1041" when it reconnects. And clients reconnect with jittered backoff, so a fleet of disconnected clients does not return in one wave.

Walking through it

Heartbeats. Send a ping every 20 to 30 seconds. If a gateway misses two or three pongs in a row, it closes the connection and removes the registry entry. The interval must also be shorter than the smallest idle timeout on the path. AWS's Application Load Balancer defaults to 60 seconds, and some mobile carrier NATs are shorter.

This WebSocket sketch uses the ws package (npm install ws). It runs as a server and prints nothing:

import { WebSocketServer, WebSocket } from "ws"

const wss = new WebSocketServer({ port: 8080 })
const alive = new WeakMap<WebSocket, boolean>()

wss.on("connection", (ws) => {
  alive.set(ws, true)
  ws.on("pong", () => alive.set(ws, true))
})

// Every 25 s: drop sockets that missed the last ping, ping the rest.
setInterval(() => {
  for (const ws of wss.clients) {
    if (!alive.get(ws)) { ws.terminate(); continue }
    alive.set(ws, false)
    ws.ping()
  }
}, 25_000)

Resume. Give every message for a user or channel an increasing sequence number. The client remembers the last one it processed:

Client last saw 1041, connection drops.
Client reconnects and sends: resume from 1041
Server replays 1042, 1043, ... from a short buffer (a Redis stream or the message store)

If the gap is too old or too large, the server tells the client to do a full resync over normal HTTP. This makes delivery at-least-once, with the client removing duplicates by sequence number. The message store stays the source of truth. A client that was offline for a day should load history over HTTP, not replay a socket.

Reconnection storms. Consider a careless deploy:

10 gateways × 100,000 connections = 1,000,000 clients
Restart all gateways at once, and every client reconnects immediately:
  1,000,000 TLS handshakes and auth checks within about a second
  the auth service normally sees about 2,000 req/s, and now sees about 1,000,000

That is a self-inflicted denial of service. Defenses:

• Drain gradually. Restart one gateway at a time. Before stopping, stop accepting new connections and tell clients to reconnect over a few minutes.
• Use jittered backoff. Each retry waits a random time up to a cap that doubles.
• Make reconnects cheap. Use resume tokens or signed session tokens, so a gateway can check a reconnect without calling the auth service.

function reconnectDelayMs(attempt: number): number {
  const cap = Math.min(60_000, 1_000 * 2 ** attempt)
  return Math.random() * cap // full jitter: a random wait below the cap
}

The cap doubles on each attempt, from 1 second up to 60 seconds. The actual wait is random below that cap. Without the randomness, every client retries in step, and the storm repeats on each wave.

Common mistakes

• Heartbeats longer than the idle timeout. The NAT drops the connection, and nobody notices until a send fails.
• No jitter in backoff. Clients retry in waves.
• Full re-authentication on every reconnect. Use a resume token instead.

Check yourself

1. Why must a heartbeat be shorter than the smallest idle timeout on the path?

   Answer: otherwise a NAT or load balancer drops the idle connection, and nobody notices until a send fails.

2. Why does jitter matter when many clients reconnect?

   Answer: without it, clients that dropped together retry together, and the storm repeats on each wave.

Key takeaways

• Heartbeats detect dead connections, and they must be shorter than any idle timeout on the path.
• Sequence numbers and resume let reconnects recover missed messages.
• Drain gateways gradually, and make reconnects cheap.
• Jittered exponential backoff spreads reconnects out and prevents a storm.

Further learning

Next, see how presence and typing indicators avoid overwhelming the same system, in presence and typing.