Machines and programs
System calls
How a program asks the kernel to read files, open sockets or allocate memory, what each request costs, and how strace shows every call a process makes.
Your program cannot read a file, open a socket, get fresh memory from the OS, or print to the terminal on its own. User code runs in an unprivileged mode where those operations are forbidden. To do them, the program makes a system call (syscall) and asks the kernel to act on its behalf.
Concept
Before you start: the kernel is the privileged part of the OS. User mode is where your program runs. Kernel mode is where the kernel runs, with full access to hardware. A syscall is the controlled switch between the two.
The big idea
To make a syscall, the program puts a syscall number and its arguments in registers, then runs a special instruction (syscall on x86-64, svc on ARM64). The CPU switches to kernel mode. The kernel checks permissions, does the work, and returns control to your program.
That switch costs roughly 100 ns to 1 µs, depending on the CPU, the call and the security mitigations in place. One call is trivial. A server that makes millions of small read and write calls a second feels it. That is why buffered I/O exists, why Linux added io_uring (which batches many operations through shared memory), and why clock_gettime is served from the vDSO, a page of kernel code mapped into your process. Reading the time never enters the kernel at all.
Walking through it: strace a hello world
On Linux, strace prints every system call a process makes. Compile a C hello world and trace it:
$ cat hello.c
#include <stdio.h>
int main(void) { printf("hello\n"); return 0; }
$ gcc -o hello hello.c
$ strace ./hello
execve("./hello", ["./hello"], 0x7ffc... /* 20 vars */) = 0
brk(NULL) = 0x55d0c5a3e000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f...
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0"..., 832) = 832
mmap(NULL, 2170256, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f...
close(3) = 0
...
write(1, "hello\n", 6) = 6
exit_group(0) = ?The output is trimmed, and addresses vary. Read it top to bottom to see the life of a process:
execvereplaces the forked shell child with your program.brkandmmapset up memory.openat,readandmmapload the C library, because this binary is dynamically linked.- Your one line of real work is a single
writeto file descriptor 1 (stdout). exit_groupends the process.
A dynamically linked C hello world makes a few dozen syscalls. Run strace -c python3 -c 'print(1)' and you will see hundreds, because the interpreter opens many files to import its standard library.
Using it on a live process
$ strace -f -e trace=network,file -p <pid>Against a misbehaving production process, this answers questions like "which config file is it reading?", "is it stuck on a connect()?" and "why is it opening 10,000 files?" in minutes.
Classic strace slows the traced process a lot. For busy production services, prefer perf trace or eBPF tools. On macOS the equivalent is dtruss, which needs System Integrity Protection relaxed, so most people run strace inside a Linux container.
Common mistakes
- Writing many tiny writes. Each one is a syscall. Buffer output and write in larger chunks.
- Logging unbuffered in a hot path. Logging often shows up as kernel time in profiles.
- Treating
straceas free. It can slow the process enough to hide the bug you are chasing.
Check yourself
- Why does a program that writes 1 byte a million times spend so much time in the kernel? Each write is a syscall, and each one pays the mode switch.
- How would you check which files a slow service opens? Attach
strace -f -e trace=file -p <pid>to it, or useperf traceif the overhead is too high.
Key takeaways
- A syscall is the only controlled way for user code to reach the kernel. Each one switches modes.
- A syscall costs roughly 100 ns to 1 µs, so batching and buffering pay off.
io_uringbatches operations, and the vDSO serves some calls, such as reading the time, without entering the kernel.straceshows exactly what a process asks the kernel for. Use lighter tools on live production traffic.
Further learning
Next, see how the kernel hands your process its own memory and why GC pauses appear in memory management.
For the process model behind these calls, see processes and threads.
Related
How code runs
From source file to running instructions. Covers the CPU's loop, compilers, interpreters and JITs, branch prediction and the system call boundary.
Processes, threads and the kernel
What the OS gives every program, and how to see it on a live Linux box. Start here for the process model.
Async I/O
Why waiting on the network should not hold a thread, how event loops and async functions handle thousands of connections, and when goroutines or thread pools fit better.