Network faults and gray failures

Data center networks are engineered to be reliable, and they still fail often. A 2011 study of Microsoft's data centers (Gill et al., SIGCOMM) found that load balancers and top-of-rack switches failed regularly, and that redundancy masked only part of those failures. Cloud networks add software layers of their own, and those misbehave too.

The faults to design for

• Packet loss. TCP retransmits hide most of it. But retransmission timeouts start around 200 ms on Linux and back off exponentially, so a lossy link can turn a 1 ms call into a 200 ms or 1 s call.
• Delay and reordering. Queues in switches and kernels can hold a packet for seconds. Two messages on different connections can arrive in either order.
• Duplication. Retries at any layer (client library, load balancer, service mesh) can deliver the same request twice.
• Partition. A set of nodes can talk among themselves but not to another set.
• Asymmetric partition. A can send to B, but B's replies to A are dropped. A thinks B is dead. B thinks everything is fine and keeps acting as leader.
• Partial partition. A can reach B and C, B can reach C, but A cannot reach C. Leader election protocols have been knocked into livelock by exactly this.

Walking through a real partition

On October 21, 2018, routine maintenance on GitHub's network caused 43 seconds of lost connectivity between a network hub on the US East Coast and the primary data center in the same area. Their MySQL failover tooling saw the primaries as unreachable and promoted replicas in the US West Coast data center. When connectivity returned, both coasts had accepted writes the other did not have. GitHub ran in a degraded state for just over 24 hours while it reconciled data.

The partition lasted 43 seconds. Recovery took about a day. Plan for recovery to take far longer than the fault.

The full account is in GitHub's October 21 post-incident analysis.

Gray failures

The hardest network faults are not clean breaks. A gray failure is a component that is degraded in a way that some observers see and others don't. A switch drops 2% of packets. A disk takes 3 seconds on 1 in 1,000 writes. A NIC runs at a tenth of its speed.

Here is why health checks miss it:

Health check: a tiny ping every second   → succeeds, node marked healthy
Real traffic: 2% of requests dropped     → timeouts, errors, slow pages

The health check and the real workload see different things. The defense is to measure health from the client's point of view. Track the error rate and latency of real requests per backend, and eject backends whose real traffic fails, even if their /health endpoint returns 200. Envoy's outlier detection works this way.

Common mistakes

• Trusting a health endpoint to prove real traffic works. It proves much less than it seems.
• Assuming a partition is clean. One-way failures are common and confuse both sides.
• Ignoring retransmission delay. A lossy link makes timeouts fire late, which makes slow calls look like outages.

Check yourself

1. Why can a lossy link turn a 1 ms call into a 1 second call? TCP retransmission timeouts start around 200 ms and back off exponentially, so a few lost packets add hundreds of milliseconds each.

2. A switch drops 2% of packets, and the health check still passes. What is the failure? A gray failure. The lightweight check gets through, while real requests time out.

Key takeaways

• The network can lose, delay, duplicate and reorder messages, and it can split in lopsided ways.
• Asymmetric partitions are dangerous because each side draws a different conclusion.
• Gray failures pass simple health checks. Measure real request outcomes per backend.

Further learning

Next, pick the assumptions a correct algorithm should make in system models.