Rollouts and graceful termination

A rolling update replaces old pods with new ones in small batches. Zero-downtime deploys depend on two things: the new pods must be ready before the old ones leave, and the old ones must stop taking traffic before they stop running. Both depend on settings that are easy to get wrong.

The big idea

Think of changing the crew on a moving train. You need the new crew on board before the old one steps off, and the old crew must stop serving the tables before they leave the kitchen. If they leave while still cooking, the customers get nothing.

Walking through it

A rolling update has two knobs:

• maxUnavailable: how many pods below the desired count can be down during the roll.
• maxSurge: how many extra pods can exist above the desired count.

The defaults, 25% for both, suit many APIs. For zero downtime with slow readiness, use maxUnavailable: 0 and maxSurge: 1. Then each new pod must be Ready before an old one is removed.

When a pod is deleted, the steps run in this order:

1. The pod is marked Terminating and removed from the Service endpoints, so new traffic stops arriving.
2. The preStop hook runs, if one is set.
3. SIGTERM is sent to the process.
4. The terminationGracePeriodSeconds clock runs, 30 seconds by default.
5. SIGKILL is sent if the process is still running.

A preStop hook helps when the endpoint removal takes a moment to reach every proxy:

lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "sleep 5"]
terminationGracePeriodSeconds: 60

The sleep 5 gives kube-proxy or the service mesh time to stop routing to the pod before the app stops accepting connections. Without it, in-flight requests can get a reset. The app must still handle SIGTERM by refusing new work, finishing current work and exiting. The details are in graceful shutdown.

Long requests need a grace period longer than the slowest one, plus mesh drain time. A 55-second upload with a 30-second grace period is killed mid-flight on every rolling update. Raise the grace period, or make the work asynchronous.

Progressive delivery tools such as Argo Rollouts and Flagger build canaries on top of Deployments. They shift a small share of traffic, watch the metrics, then promote or abort. Even without them, kubectl rollout undo deployment/checkout returns to the previous revision. Know that command before you need it at 2 a.m.

Common mistakes

• Ignoring SIGTERM. Every deploy ends in SIGKILL, and requests are dropped.
• A grace period shorter than the longest request. Long requests die on every roll.
• No maxUnavailable: 0 on a slow-starting service. The roll can take healthy pods down before their replacements are ready.

Check yourself

1. With maxUnavailable: 0 and maxSurge: 1, when is an old pod removed? After a new pod is Ready. The service keeps its full count the whole time.
2. A 55-second request meets a 30-second grace period during a roll. What happens? SIGKILL ends the process at 30 seconds, and the request is cut off.

Key takeaways

• Use maxUnavailable: 0 and maxSurge: 1 when readiness is slow.
• Deletion order: remove from endpoints, preStop, SIGTERM, grace period, SIGKILL.
• Set the grace period longer than your slowest request.

Further learning

Canary and blue-green strategies are in deployment strategies. The signal handling in the app is in graceful shutdown.