Health checks and draining

A health check is a test that decides whether a backend should get traffic. Active checks probe each backend on a schedule. Passive checks watch the real responses from live traffic. Draining is how you take a backend out on purpose without cutting off the requests it is already handling.

The big idea

Active checks catch servers that are down even when no traffic is flowing, but they are slow. Passive checks react within a few requests and catch problems a probe misses, such as one bad dependency on a single code path. Use both.

Walking through it

An active check calls GET /healthz every 5 seconds. A backend is marked unhealthy after 3 failed checks in a row, and healthy again after 2 passes.

t=0s    server dies
t=5s    check 1 fails  (1 of 3)
t=10s   check 2 fails  (2 of 3)
t=15s   check 3 fails  (3 of 3) -> ejected

So a dead server can still get traffic for about 15 seconds. Passive checks shorten that. Envoy's defaults eject a host after 5 consecutive 5xx responses, keep it out for 30 seconds (longer on each repeat), and never eject more than 10% of the pool at once. The Envoy documentation lists every setting.

Design the health endpoint carefully

• Liveness asks "is the process alive?" It should be trivial. Kubernetes uses it to restart containers, so a liveness check that calls the database can cause restart loops.
• Readiness asks "should this instance get traffic right now?" It should reflect whether the instance can serve, with warm caches and a working connection pool. Balancers should use readiness.
• Don't check shared dependencies deeply. If every instance's check calls the same database, a 2-second database blip fails every check at once. The balancer removes the whole fleet, and a 2-second hiccup becomes a 2-minute outage.

To guard against that last case, balancers fail open. Envoy's panic threshold (default 50%) means that when more than half the pool looks unhealthy, it ignores health status and spreads traffic across everyone. The checks are more likely wrong than half the fleet being broken.

Draining on purpose

When you replace an instance, the balancer should stop sending it new requests and let in-flight ones finish. AWS calls this deregistration delay, with a default of 300 seconds. Your service must cooperate: on SIGTERM, fail readiness, keep serving for the drain window, then exit.

Slow start is the mirror image. A fresh instance has cold caches and an unwarmed JIT compiler, so it should ramp up over 30 to 60 seconds. Least-request balancing makes this worse. A new instance has zero requests in flight, so every balancer sends it work at once.

Common mistakes

• Deep health checks on shared dependencies. One database blip can remove the whole fleet. Keep checks shallow and enable fail open.
• Skipping graceful shutdown. Rolling deploys without draining drop requests on every deploy. Run a load test during a rolling deploy and look for 5xx spikes.
• Using liveness to test dependencies. A slow database then becomes a restart storm.

Check yourself

1. With a check every 5 seconds and ejection after 3 failures, how long until a dead server leaves the pool?

   About 15 seconds, at the third failed check.

2. Why does the balancer keep traffic flowing when more than half the pool looks unhealthy?

   The health checks are more likely wrong than half the fleet being broken, so ignoring the status spreads traffic across everything.

Key takeaways

• Active checks find servers that are down. Passive checks find real failures on live traffic. Use both.
• Readiness decides who gets traffic. Liveness only decides whether to restart. Keep both shallow.
• A shared dependency in every health check can empty the whole pool at once.
• Draining lets in-flight requests finish, and slow start stops a new instance from being swamped.

Further learning

Next: service discovery and meshes, which covers how the balancer learns which instances exist at all.