Profiling with perf and eBPF

Profiling answers a narrower question than the checklists. The checklists tell you that the CPU is saturated and which process is burning it. A profiler tells you which functions inside that process are doing the work. It does this without changing the code, by taking samples of the call stack many times per second.

The big idea

Imagine a photo of a busy kitchen taken every few milliseconds. If the grill cook appears in 60% of the photos, the grill is where the time goes. A profiler does the same thing with the program's call stack. The more often a function appears on the stack, the more CPU time it costs.

Walking through it

The main tools:

• perf top shows the hottest functions live. perf record -g followed by perf report saves a sample for later. The output needs symbols, either debug info or frame pointers, to show readable stacks. In containers, profile from the host with care about process namespaces.
• Flame graphs turn those samples into a picture. Each bar is a function, and its width is the share of samples where it appeared. Wide bars near the top are the places to look. A flame graph of time spent on the CPU finds hot code. Time spent waiting needs a different trace, because a sleeping thread is not on the CPU.
• eBPF tools such as biolatency, runqlat, execsnoop and tcpconnect, from the bcc and bpftrace packages, answer specific questions at low overhead. They are the safer choice for production when you cannot run perf widely.

Here is a sensible order. Confirm the CPU is saturated with pidstat. Record a short perf sample of that one process. Read the widest stacks. If the process is mostly waiting, not burning CPU, use an off-CPU view or an eBPF latency tool instead.

Common mistakes

• Profiling before you confirm the CPU is the bottleneck. A flame graph of an I/O-bound process points at code that is not the problem.
• Reading only the widest bar. Check the call stacks above it to see who called the hot function.
• Running wide tracing in the middle of an incident without a limit. Record for a short window, and read the result afterward.

Check yourself

1. A process is at 5% CPU, but requests are slow. Should you start with a CPU flame graph? Probably not. The process is mostly waiting, so use an off-CPU view or an eBPF latency tool.
2. What does a wide bar in a flame graph mean? That function appeared in a large share of the samples, so it used a large share of the CPU time.

Key takeaways

• Profile after you confirm the resource. Profiling answers "which code" and the checklists answer "which resource".
• Flame graphs show time as width. Read the widest stacks first.
• Use eBPF tools for low-overhead answers in production.

Further learning

The profiling mechanics are in flame graphs and CPU profiling. For the Gregg reference on flame graphs, see below.

Flame GraphsHomepage for Flame Graphs: a visualization for stack traces.brendangregg.com