Guide

Linux and networking

Processes and signals

What a process is, how to read its state, and how signals and exit codes tell you why it stopped. Exit 137 and 143 come up constantly.

2 of 162 · 3 min read

Every program you run becomes a process: a running copy with its own memory, open files and a number called its PID (process ID). Each process has a parent. PID 1 is the init system, usually systemd, or your app itself inside a container. When something goes wrong with a service, the process table and the process's exit code are the first clues.

Concept

A signal is a short message the kernel delivers to a process, such as "please stop" or "you did something illegal". The process can often catch it and react. The most common ones are listed below.

Walking through it

Start with the process table. This command lists the biggest memory users, with how long each has been running:

Bash
ps -eo pid,ppid,stat,etime,rss,cmd --sort=-rss | head -5

The STAT column tells you what a process is doing:

  • R: running or ready to run.
  • S: sleeping, waiting for something such as a network reply.
  • D: uninterruptible sleep, usually waiting on disk or NFS. It cannot be killed until that I/O finishes.
  • Z: a zombie. It has exited, but its parent has not collected the result.
  • T: stopped.

Now the signals you will use most:

SignalNumberDefaultTypical use
SIGTERM15Terminate"Please shut down." The process can catch it and clean up.
SIGINT2TerminateCtrl-C in a terminal
SIGKILL9TerminateCannot be caught. The kernel removes the process at once.
SIGHUP1TerminateTerminal hung up. Many daemons reload config on it.
SIGQUIT3Core dumpMakes the JVM and Go print thread or goroutine dumps

When a process dies from signal N, the shell and container runtime report exit code 128 + N. Two codes to memorize:

  • 137 = 128 + 9, so the process was SIGKILLed. In containers this usually means the kernel's out-of-memory killer fired.
  • 143 = 128 + 15, so the process was stopped by SIGTERM.

You will see both in kubectl describe pod and in systemctl status.

Common mistakes

  • Using SIGKILL as the normal way to stop a service. It cannot be caught, so no cleanup code runs and in-flight work is lost.
  • Reading 137 as a crash in your code. It usually means a memory limit was hit. Check the limit first.
  • Running kill -9 on a process in D state. It does nothing until the disk wait ends. Fix the storage problem instead.

Check yourself

  1. A container exits with code 137. Which signal was it, and what is the likely cause? SIGKILL (128 + 9). Most often the kernel killed it after it hit its memory limit.
  2. What does exit code 143 tell you? The process was stopped by SIGTERM (128 + 15). That is a normal stop request, so check whether the service handled it.

Key takeaways

  • A process has a PID, a parent and a state. The STAT column shows whether it runs, sleeps, or waits on disk.
  • Signals are requests. SIGTERM can be handled, SIGKILL cannot.
  • Exit code 128 + N names the signal N. 137 is SIGKILL, and 143 is SIGTERM.

Further learning

The process model in more depth is in processes and threads. Next, see how a service should handle SIGTERM in graceful shutdown.

Signals and graceful shutdown

Linux for engineers

The parts of Linux you touch in production, from shell pipelines and systemd services to permissions, disks, network debugging commands and safe remote access.

Graceful shutdown

How orchestrators stop a process with SIGTERM, then SIGKILL after a grace period, and how a service drains in-flight work inside that window.

Kubernetes in production

Requests and limits, probes, autoscaling, disruption budgets, packaging and the mistakes that cause outages.