Graceful shutdown

When a platform stops a service, it does not kill it at once. It sends SIGTERM, which means "please finish up". It waits a fixed grace period. Only then does it send SIGKILL, which cannot be caught. A service that handles SIGTERM can stop taking new work, finish the requests already running, and exit cleanly. One that ignores SIGTERM loses those requests on every deploy.

The big idea

Think of a restaurant closing. The host stops seating new guests, the kitchen finishes the orders already placed, and then the lights go off. SIGKILL is locking the door with people still inside. The restaurant can close quickly, but nobody gets their meal.

The analogy has a limit. A real server also needs to be removed from the load balancer before it stops, which is a separate race. That part is covered in rollouts and graceful termination.

Walking through it

A Go service that drains for 25 seconds. This is a sketch: the server setup is left out.

ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM, syscall.SIGINT)
defer stop()

go func() { _ = srv.ListenAndServe() }()

<-ctx.Done() // SIGTERM received
shutdownCtx, cancel := context.WithTimeout(context.Background(), 25*time.Second)
defer cancel()
_ = srv.Shutdown(shutdownCtx) // stop accepting, finish in-flight requests

Trace one shutdown, using the 30-second Kubernetes default:

1. Kubernetes sends SIGTERM at t = 0 and starts the 30-second grace period.
2. Shutdown stops new connections and waits for in-flight requests.
3. If the slowest request finishes by t = 12 s, the process exits cleanly at that moment.
4. If a request is still running at t = 25 s, the context expires and the process exits anyway.
5. If the process were still alive at t = 30 s, SIGKILL would end it.

The 25-second drain sits inside the 30-second default on purpose. A drain longer than the grace period is cut off by SIGKILL anyway.

Common mistakes

• Using the shell form of CMD. CMD python app.py runs the app under /bin/sh, which is PID 1. The shell does not forward SIGTERM, so the app gets SIGKILL with no warning. Use CMD ["python", "app.py"], or an init such as tini.
• Draining longer than the grace period. Raise the grace period, or make the work asynchronous.
• Ignoring SIGTERM. Every deploy then ends in a SIGKILL and drops requests.

Check yourself

1. The grace period is 30 seconds, but draining takes 45 seconds. What happens at 30 seconds? SIGKILL ends the process, and the requests still in flight are cut off.
2. Why does a container started with CMD python app.py ignore a stop request? The shell is PID 1 and does not pass SIGTERM on to the app.

Key takeaways

• Stopping is SIGTERM, then a grace period, then SIGKILL.
• Handle SIGTERM by refusing new work, finishing current work, and exiting within the grace period.
• Use exec-form CMD or an init process, so the signal reaches your app.

Further learning

The signal basics are in processes and signals. For how this interacts with traffic during a deploy, see rollouts and graceful termination. The deeper mechanism is in signals and graceful shutdown.