Timeouts and failure detectors

Since you can't know whether a node is dead, you guess. If no response arrives within T, you declare the request or the node failed. The guess is only as good as the value of T, and choosing it is a trade-off between two kinds of error.

The big idea

• Too short. You declare live nodes dead. In a request path, you retry calls that would have succeeded, which adds load. In a cluster, you trigger failovers, which are risky in themselves.
• Too long. You wait a long time before reacting to real failures. Users see a frozen request. A dead leader stays in place.

A node under heavy load can be declared dead, so its work moves to others. Those others become overloaded, miss their own deadlines and are declared dead in turn. That is a cascading failure.

Walking through a timeout choice

Suppose a downstream inventory service has p50 = 8 ms, p99 = 45 ms, p99.9 = 120 ms, and occasional 2-second outliers during garbage collection. Your endpoint has a 300 ms budget.

Better still is a hedged request. At about 50 ms, if the first call hasn't answered, send a second copy to another replica. Only about 1% of requests trigger the hedge, so the tail shrinks without much extra load. See availability, latency and the tail.

The practical rule is to measure. Set each timeout from the observed latency distribution of the dependency, not from a guess.

Heartbeats and phi accrual

For cluster membership, nodes send periodic heartbeats. A fixed rule suspects a node when heartbeats stop for longer than a set time. That rule ignores context. Two seconds of silence is alarming from a node that normally heartbeats every 100 ms. It is normal from a node on a congested cross-region link.

The phi accrual failure detector (Hayashibara et al., 2004) outputs a suspicion level instead of a yes or no. It keeps a sliding window of recent arrival times and models them as a distribution:

phi = -log10( probability the heartbeat is only late, given the history )

phi = 1  → about a 10% chance the node is alive and just late
phi = 2  → about a 1% chance
phi = 3  → about a 0.1% chance

Each application picks its own threshold. Cassandra uses phi accrual with a default phi_convict_threshold of 8, and Akka Cluster uses it too. The detector adapts to the network. On a jittery link, it waits longer before it convicts a node.

Common mistakes

• Choosing a timeout by guess. Measure p99 and p99.9 for the dependency first.
• Retrying past the time budget. A retry that can't finish within the caller's budget only adds load.
• Using a fixed heartbeat timeout on a jittery link. Use a detector that learns the link's normal behavior.

Check yourself

1. With p99 at 45 ms and a 300 ms budget, why not set a 2-second timeout? One slow call would consume the whole budget and then some, so the caller misses its own latency target.

2. What does phi = 2 mean? By the detector's history, there is about a 1% chance that the node is alive and just late.

Key takeaways

• A timeout is a guess about whether a node is dead. Pick it from measured latency.
• Too short causes false failures and cascades. Too long causes slow reactions.
• Hedged requests cut the tail without much extra load.
• Phi accrual replaces a yes or no with a level that adapts to the network.

Further learning

Next, see how to make a lock safe when its holder pauses in fencing tokens.

Timeouts, retries, and backoff with jitter (AWS Builders' Library)How Amazon uses timeouts, retries, and exponential backoff with jitter to handle transient failures without overwhelming struggling services.aws.amazon.com