Signals and graceful shutdown

A signal is a short notification the kernel delivers to a process, such as "stop now" or "a child exited". Signals are how the outside world tells your service to shut down. If your service handles them well, a deploy finishes its in-flight requests. If it doesn't, those requests fail halfway through.

The big idea

The signal you care about most is SIGTERM (number 15). It is a polite request to stop. Your program can catch it, finish its work and exit. SIGKILL (number 9) is different. The kernel destroys the process, and your program never sees the signal. There is no cleanup, no flushing and no "finish the current request".

Walking through a Kubernetes shutdown

When Kubernetes deletes a pod, the steps run roughly in this order.

Removing the endpoint and sending SIGTERM happen at about the same time. A few requests may still arrive just after SIGTERM. Many teams add a short preStop sleep of 5 to 10 seconds so load balancers catch up before the app stops listening.

Here is the shape of a graceful shutdown in Go. This is a sketch: mux and db come from your app.

srv := &http.Server{Addr: ":8080", Handler: mux}
go srv.ListenAndServe()

stop := make(chan os.Signal, 1)
signal.Notify(stop, syscall.SIGTERM, syscall.SIGINT)
<-stop

ctx, cancel := context.WithTimeout(context.Background(), 25*time.Second)
defer cancel()
srv.Shutdown(ctx) // stop listening, wait for in-flight requests
db.Close()

The 25-second timeout is shorter than Kubernetes' 30-second grace period. The process then exits on its own terms, not because SIGKILL arrived.

The PID 1 trap in containers

Inside a container, your entrypoint is PID 1. The kernel treats PID 1 specially. Signals whose default action is "terminate" are ignored unless the process has installed a handler.

So a Node or Python script that never registers a SIGTERM handler ignores docker stop. Docker waits 10 seconds, then sends SIGKILL. Every deploy then takes 10 extra seconds per container, and in-flight work is cut off.

Two other mistakes cause the same trouble.

• Shell-form entrypoints. CMD npm start runs /bin/sh -c "npm start". The shell is PID 1. It may not forward SIGTERM to your app. Use exec form, CMD ["node", "server.js"], or exec in a wrapper script.
• No reaper. If your app spawns child processes and PID 1 never calls wait(), zombies pile up. Use a small init such as tini (docker run --init adds one).

Common mistakes

• Ignoring SIGTERM. The process gets killed mid-request after the grace period. Install a handler that drains and exits.
• Finishing old work while still taking new work. Stop accepting new requests first, then finish what is in flight.
• Setting a shutdown timeout longer than the grace period. The kernel's SIGKILL wins, and your cleanup never runs.

Check yourself

1. Why can't a program clean up after SIGKILL? The kernel destroys the process directly. The program never gets to run handler code.
2. Your container takes 10 seconds to stop on every deploy. What is the likely cause? PID 1 ignores SIGTERM because no handler is installed, so Docker waits for the timeout and then sends SIGKILL.

Key takeaways

• SIGTERM is a request your program can catch. SIGKILL is not delivered to your program at all.
• A graceful shutdown stops accepting work, finishes in-flight requests, closes resources and exits before the grace period.
• In containers, PID 1 ignores SIGTERM unless a handler is installed. Use exec-form CMD and a handler.
• A process that stops late is usually waiting for the kernel's SIGKILL. Measure how long shutdown takes.

Further learning

The Kubernetes docs cover the exact pod termination order. The signal(7) man page lists every signal and its default action.

Kubernetes: Pod terminationThis page describes the lifecycle of a Pod. Pods follow a defined lifecycle, starting in the Pending phase, moving through Running if at least one of its primary containers starts OK, and then through either the Succe…Kubernetes