Health probes

Probes are how the kubelet asks a container whether it is alive and ready for traffic. There are three, and each one does a different thing when it fails. Mixing them up is one of the most common ways to turn a small outage into a large one.

The big idea

Think of a shop. Startup is the period when the staff are still unpacking the stock, so nobody is judged yet. Readiness is whether the shop is open to customers. If it fails, the shop stops taking visitors but stays open. Liveness is whether the building is still standing. If it fails, you rebuild the shop. You rebuild only when it is truly wedged.

Walking through it

startupProbe:
  httpGet: { path: /livez, port: 8080 }
  failureThreshold: 30
  periodSeconds: 5          # up to 150 s to start
readinessProbe:
  httpGet: { path: /readyz, port: 8080 }
  periodSeconds: 5
livenessProbe:
  httpGet: { path: /livez, port: 8080 }
  periodSeconds: 10
  failureThreshold: 3

The startup probe allows up to 30 tries at 5 seconds each, which is 150 seconds of boot time. Until it passes, liveness does not run, so a slow boot is not mistaken for a hang.

Now the deep-dependency trap. Imagine twelve replicas that all check the database in their liveness probe. The database times out for 20 seconds. Every liveness check fails. Kubernetes restarts all twelve together. Each one reopens its connection pool, against a database that is still recovering. The outage lasts longer than the database blip did.

Put the same check on readiness instead, and the result is correct. The pods are removed from the endpoints for 20 seconds, the load balancer returns 503 during that window, and the pods return when the database recovers. That is fail closed, without a restart storm.

Two more rules:

• Set timeouts with headroom. A timeoutSeconds: 1 on a busy endpoint fails under normal load.
• Avoid heavy probes. A /health endpoint that runs queries adds load of its own. Prefer httpGet or the native gRPC health protocol over exec probes, which fork a process every period.

Readiness also gates rollouts. A Deployment does not remove old pods until new ones are ready. A readiness probe that never passes stalls the rollout forever. kubectl rollout status shows where it is stuck.

Common mistakes

• Deep checks on liveness. A dependency blip becomes a restart storm.
• No startup probe on a slow boot. Liveness kills the container during initialization.
• Aggressive timeouts. Fast probes fail under real load and cause false restarts.

Check yourself

1. The database is slow for 20 seconds. The database check is on readiness. What happens to traffic? The pods leave the endpoints for that window, and the load balancer returns 503. They come back when the database recovers. No pods restart.
2. Why does a slow-booting JVM need a startup probe? Without it, liveness starts checking during boot and restarts the container before it finishes initializing.

Key takeaways

• Readiness decides traffic. Liveness decides restarts. Never mix them.
• Keep liveness shallow, so it only fires for a wedged process.
• Use a startup probe for slow boots, and give probes timeouts with headroom.

Further learning

The rollout side of readiness is in rollouts and graceful termination. The process signals that end a container are in graceful shutdown.

Configure Liveness, Readiness and Startup ProbesThis page shows how to configure liveness, readiness and startup probes for containers. For more information about probes, see Liveness, Readiness and Startup Probes. Before you beginYou need to have a Kubernetes clus…Kubernetes