Guide

Almost everything you deploy runs on Linux. A container is a Linux process with some isolation around it. A Kubernetes node is a Linux VM. A managed database is Postgres on a Linux box someone else patches. When something breaks at the edge of your code, such as a port that won't bind, a disk that is "full" with 40% free, or a service that ignores Ctrl-C, the explanation is usually a Linux concept.

You do not need to be a kernel expert. You need a working toolkit: enough shell to slice logs and automate chores, enough process and service knowledge to run and stop services cleanly, enough permissions and disk knowledge to diagnose "permission denied" and "no space left", and enough network commands to answer "can A reach B, and if not, which layer fails?"

Walking through the toolkit

The eight pages below follow the order you need them in. The first four are the basics. The last four are the practices that use them.

StepConceptWhat it answers
1Shell pipes and streamsHow do small tools chain into one answer?
2Processes and signalsWhat is running, and why did it stop?
3Permissions and capabilitiesWho is allowed to read or enter this path?
4Disk space and mountsWhy is the disk "full" when df looks fine?
5Safe bash scriptsHow do I write a script that fails loudly?
6Graceful shutdownHow does a service stop without dropping requests?
7systemd and journaldHow does a service run, restart and log on a VM?
8Debugging connectivityWhich layer breaks, and how do I reach a private host?

Images, not servers

Most teams have stopped editing servers by hand. A mutable server is configured in place, over SSH or with Ansible. Its state drifts over time, and rollback means hoping the configuration converges. An immutable setup bakes an image in CI, such as a container image or an AMI, and replaces servers to change them. Every instance from one image is identical, and rollback means redeploying the previous image.

The industry summary is cattle, not pets: instances are numbered and replaceable, not named and nursed. Long-lived hosts such as database servers may still need configuration management. If you find yourself hand-editing /etc on a production web server, the fix belongs in the image or the Terraform code, not only on that box. Fixing the box by hand is a time bomb, because the next deploy erases it.

Common mistakes

  • Writing scripts without guards. An unset variable can turn a cleanup into a delete of everything. Start every script with the safe preamble.
  • Guessing at connectivity. Check each layer in order and bring the output to the network team.
  • Treating "disk full" as one problem. It can be blocks, inodes, or deleted files still open.
  • Patching production by hand without fixing the source. The next deploy will undo the patch at the worst moment.

Key takeaways

  • Most production Linux problems map to a concept: a signal, a permission bit, a full inode table, or a layer of the network path.
  • Pipes, exit codes and set -euo pipefail make command-line work reliable.
  • Services run under systemd, stop by SIGTERM and a grace period, and log to the journal.
  • Prefer immutable images and session managers over hand-configured servers and open SSH ports.

Further learning

Start with the first step in the table, then the page on processes and signals, which the service pages build on.

MIT's Missing Semester is the best free introduction to the shell. The scripting and data-wrangling lectures cover most of the tools above.

Lecture 2: Shell Tools and Scripting (2020)Missing Semester
The Missing Semester of Your CS Education (MIT)Master powerful tools that will make you a more productive computer scientist and programmer.Missing Semester The Linux Command Line (William Shotts)Linux Command Line Books by William Shottslinuxcommand.org AWS Systems Manager Session ManagerManage your nodes using a secure browser-based interactive shell or the AWS CLI without having to open inbound ports.docs.aws.amazon.com