Linux and networking
Linux for engineers
The parts of Linux you touch in production, from shell pipelines and systemd services to permissions, disks, network debugging commands and safe remote access.
Almost everything you deploy runs on Linux. A container is a Linux process with some isolation around it. A Kubernetes node is a Linux VM. A managed database is Postgres on a Linux box someone else patches. When something breaks at the edge of your code, such as a port that won't bind, a disk that is "full" with 40% free, or a service that ignores Ctrl-C, the explanation is usually a Linux concept.
You do not need to be a kernel expert. You need a working toolkit: enough shell to slice logs and automate chores, enough process and service knowledge to run and stop services cleanly, enough permissions and disk knowledge to diagnose "permission denied" and "no space left", and enough network commands to answer "can A reach B, and if not, which layer fails?"
Walking through the toolkit
The eight pages below follow the order you need them in. The first four are the basics. The last four are the practices that use them.
| Step | Concept | What it answers |
|---|---|---|
| 1 | Shell pipes and streams | How do small tools chain into one answer? |
| 2 | Processes and signals | What is running, and why did it stop? |
| 3 | Permissions and capabilities | Who is allowed to read or enter this path? |
| 4 | Disk space and mounts | Why is the disk "full" when df looks fine? |
| 5 | Safe bash scripts | How do I write a script that fails loudly? |
| 6 | Graceful shutdown | How does a service stop without dropping requests? |
| 7 | systemd and journald | How does a service run, restart and log on a VM? |
| 8 | Debugging connectivity | Which layer breaks, and how do I reach a private host? |
Images, not servers
Most teams have stopped editing servers by hand. A mutable server is configured in place, over SSH or with Ansible. Its state drifts over time, and rollback means hoping the configuration converges. An immutable setup bakes an image in CI, such as a container image or an AMI, and replaces servers to change them. Every instance from one image is identical, and rollback means redeploying the previous image.
The industry summary is cattle, not pets: instances are numbered and replaceable, not named and nursed. Long-lived hosts such as database servers may still need configuration management. If you find yourself hand-editing /etc on a production web server, the fix belongs in the image or the Terraform code, not only on that box. Fixing the box by hand is a time bomb, because the next deploy erases it.
Common mistakes
- Writing scripts without guards. An unset variable can turn a cleanup into a delete of everything. Start every script with the safe preamble.
- Guessing at connectivity. Check each layer in order and bring the output to the network team.
- Treating "disk full" as one problem. It can be blocks, inodes, or deleted files still open.
- Patching production by hand without fixing the source. The next deploy will undo the patch at the worst moment.
Key takeaways
- Most production Linux problems map to a concept: a signal, a permission bit, a full inode table, or a layer of the network path.
- Pipes, exit codes and
set -euo pipefailmake command-line work reliable. - Services run under systemd, stop by SIGTERM and a grace period, and log to the journal.
- Prefer immutable images and session managers over hand-configured servers and open SSH ports.
Further learning
Start with the first step in the table, then the page on processes and signals, which the service pages build on.
MIT's Missing Semester is the best free introduction to the shell. The scripting and data-wrangling lectures cover most of the tools above.
Related
Shell pipes and streams
How programs talk through stdin, stdout and stderr, and how pipes chain small tools into one answer to a production question.
Processes and signals
What a process is, how to read its state, and how signals and exit codes tell you why it stopped. Exit 137 and 143 come up constantly.
Debugging connectivity
When A cannot reach B, check each layer from the bottom up and write down what each one said. Timeouts and refusals point to different causes.
Containers from the inside
Namespaces, cgroups and layered images - what a container really is, how images are built and shipped, and how to write small, fast, secure Dockerfiles.