Guide

Your program cannot read a file, open a socket, get fresh memory from the OS, or print to the terminal on its own. User code runs in an unprivileged mode where those operations are forbidden. To do them, the program makes a system call (syscall) and asks the kernel to act on its behalf.

Concept

Before you start: the kernel is the privileged part of the OS. User mode is where your program runs. Kernel mode is where the kernel runs, with full access to hardware. A syscall is the controlled switch between the two.

The big idea

To make a syscall, the program puts a syscall number and its arguments in registers, then runs a special instruction (syscall on x86-64, svc on ARM64). The CPU switches to kernel mode. The kernel checks permissions, does the work, and returns control to your program.

That switch costs roughly 100 ns to 1 µs, depending on the CPU, the call and the security mitigations in place. One call is trivial. A server that makes millions of small read and write calls a second feels it. That is why buffered I/O exists, why Linux added io_uring (which batches many operations through shared memory), and why clock_gettime is served from the vDSO, a page of kernel code mapped into your process. Reading the time never enters the kernel at all.

Walking through it: strace a hello world

On Linux, strace prints every system call a process makes. Compile a C hello world and trace it:

Bash
$ cat hello.c
#include <stdio.h>
int main(void) { printf("hello\n"); return 0; }
$ gcc -o hello hello.c
$ strace ./hello
execve("./hello", ["./hello"], 0x7ffc... /* 20 vars */) = 0
brk(NULL)                               = 0x55d0c5a3e000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f...
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0"..., 832) = 832
mmap(NULL, 2170256, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f...
close(3)                                = 0
...
write(1, "hello\n", 6)                  = 6
exit_group(0)                           = ?

The output is trimmed, and addresses vary. Read it top to bottom to see the life of a process:

  • execve replaces the forked shell child with your program.
  • brk and mmap set up memory.
  • openat, read and mmap load the C library, because this binary is dynamically linked.
  • Your one line of real work is a single write to file descriptor 1 (stdout).
  • exit_group ends the process.

A dynamically linked C hello world makes a few dozen syscalls. Run strace -c python3 -c 'print(1)' and you will see hundreds, because the interpreter opens many files to import its standard library.

Using it on a live process

Bash
$ strace -f -e trace=network,file -p <pid>

Against a misbehaving production process, this answers questions like "which config file is it reading?", "is it stuck on a connect()?" and "why is it opening 10,000 files?" in minutes.

Classic strace slows the traced process a lot. For busy production services, prefer perf trace or eBPF tools. On macOS the equivalent is dtruss, which needs System Integrity Protection relaxed, so most people run strace inside a Linux container.

Common mistakes

  • Writing many tiny writes. Each one is a syscall. Buffer output and write in larger chunks.
  • Logging unbuffered in a hot path. Logging often shows up as kernel time in profiles.
  • Treating strace as free. It can slow the process enough to hide the bug you are chasing.

Check yourself

  • Why does a program that writes 1 byte a million times spend so much time in the kernel? Each write is a syscall, and each one pays the mode switch.
  • How would you check which files a slow service opens? Attach strace -f -e trace=file -p <pid> to it, or use perf trace if the overhead is too high.

Key takeaways

  • A syscall is the only controlled way for user code to reach the kernel. Each one switches modes.
  • A syscall costs roughly 100 ns to 1 µs, so batching and buffering pay off.
  • io_uring batches operations, and the vDSO serves some calls, such as reading the time, without entering the kernel.
  • strace shows exactly what a process asks the kernel for. Use lighter tools on live production traffic.

Further learning

Next, see how the kernel hands your process its own memory and why GC pauses appear in memory management.

For the process model behind these calls, see processes and threads.