Shell pipes and streams

Most Linux tools do one small job and hand their results to the next tool. They talk through three streams: stdin (input), stdout (normal output) and stderr (error messages). Every program also ends with an exit code: 0 means success, anything else means failure. A pipe, written |, connects one program's stdout to the next program's stdin.

Walking through it

Say an incident is in progress and you need to know which client IPs are getting server errors. The nginx access log has the answer on each line.

awk '$9 >= 500 {print $1}' /var/log/nginx/access.log \
  | sort | uniq -c | sort -rn | head -10

Each stage does one step:

• awk keeps lines whose field 9 (the status code) is 500 or higher, and prints field 1 (the client IP).
• sort puts identical IPs next to each other.
• uniq -c counts each group. It only merges adjacent lines, which is why the sort comes first.
• sort -rn orders the counts from highest to lowest.
• head -10 keeps the top ten.

The output has one line per IP. Addresses below are invented to show the shape:

    412 203.0.113.9
     87 198.51.100.23
     12 192.0.2.44

Redirection in one table

The text tools

• grep finds lines. grep -rn "timeout" src/ searches folders and prints line numbers.
• awk works on columns. awk -F, '{sum += $3} END {print sum}' sales.csv adds up the third CSV column.
• sed edits streams. sed -n '100,120p' file prints lines 100 to 120.
• jq does the same job for JSON. jq -r 'select(.level == "error") | .ts[0:16]' app.log | sort | uniq -c counts errors per minute, if each line is a JSON log record with an ISO timestamp.
• find and xargs handle files in bulk. Use -print0 and -0 so filenames with spaces don't break the pipeline.

Common mistakes

• Skipping the sort before uniq -c. The counts look wrong because equal lines weren't adjacent.
• Using sed -i 's/old/new/' on macOS. BSD sed needs sed -i ''. The GNU form fails with a confusing error.
• Forgetting -print0 with xargs. A file named my log.txt splits into two arguments.
• Trusting a pipeline that failed halfway. Only the last stage's exit code counts unless you set pipefail. See safe bash scripts.

Check yourself

1. Why does uniq -c need sort first? Because uniq only merges lines that sit next to each other.
2. In cmd > out.log 2>&1, where does an error message end up? In out.log, because 2>&1 sends stderr to wherever stdout goes.

Key takeaways

• Small tools connected by pipes answer most log questions in one line.
• Stdout is for results, stderr is for errors, and the exit code is the signal scripts read.
• sort before uniq, and -print0 before xargs -0, prevent the two most common wrong answers.

Further learning

Next, learn how these programs run as processes and how signals stop them, in processes and signals.

The Missing Semester of Your CS Education (MIT)Master powerful tools that will make you a more productive computer scientist and programmer.Missing Semester

ShellCheckShellCheck finds bugs in your shell scriptsshellcheck.net