Linux and networking
Memory, page cache and OOM
Why "free" memory looking low is normal, how to read available memory, and why swap and OOM kills are the two signals that memory is actually short.
Linux uses spare RAM as page cache: copies of file contents kept in memory, so repeat reads are fast. That means free will often show little unused memory, and that is normal. The number that matters is available, which estimates how much memory can be reclaimed without swapping. Two signals show real memory shortage: the box is swapping, or the kernel is killing processes.
The big idea
Think of a desk. Cache is the reference books you keep open because you use them often. They fill the desk, but you can clear them away the moment you need the space. Available is the desk space you could free up. A desk full of books is not a problem. A desk where you are constantly shuffling papers to the floor is.
Walking through it
Read the table as "symptom, then what it means":
| Symptom | Likely meaning |
|---|---|
available is low, and si or so in vmstat is non-zero | Memory pressure. The box is swapping. |
Major page faults are high (sar -B) | Reading from disk to satisfy memory requests |
dmesg shows an OOM kill, or the container exits with 137 | A process hit its limit, or the host ran out of memory |
| Cache drops after a large sequential read | A one-time read pushed out hot data |
Here is a concrete case. A database holds 60 GB of cache on a 64 GB box. free shows almost nothing free. This is healthy. The database is using memory well. Killing it "to free memory" would throw away the cache and make every query slow again.
The opposite case is a service whose RSS (the memory the process itself holds) keeps growing, with available shrinking and so (swap out) rising. That is real pressure. Fail-fast with an OOM kill and restart is often better than silent swap, because swap makes p99 latency collapse long before anything crashes.
Warning
Confusing RSS with page cache causes false alarms. Check whether the memory belongs to the process (RSS) or to the cache, and use available as the headline number.
Common mistakes
- Reading
freeas "memory used up". Useavailable. - Keeping swap on a latency-sensitive service without thinking about it. Once it swaps, p99 usually collapses. Check your platform's default.
- Treating exit 137 as a bug in the code first. Check the memory limit and the node's memory first.
Check yourself
freeshows 2 GB free on a 64 GB box, andavailableshows 40 GB. Is there a problem? No. The memory is cache, and 40 GB is still available to hand out.- A pod is exiting with 137 while the node has plenty of free RAM. Why? The pod hit its cgroup memory limit. The node's free memory does not matter for that limit.
Key takeaways
- Page cache fills idle RAM on purpose. Low
freeis not a problem by itself. - Trust
available, notfree. - Real shortage shows up as swapping and OOM kills (exit 137).
Further learning
The virtual memory system is in virtual memory. Memory leaks and kills are covered in memory leaks and OOM kills. For how containers apply memory limits, see container throttling.
The USE Method (Brendan Gregg)The USE Methodbrendangregg.comRelated
Linux performance analysis
The USE method and the 60-second checklist for finding why a box is slow, covering CPU, memory, disk, network and the container limits that look like application bugs.
The USE method
A checklist for finding a bottleneck: for every resource, check utilization, saturation and errors before you guess at the application.
Memory leaks and OOM kills
What a leak looks like in a garbage-collected language, how to find one by diffing heap profiles, why process memory is bigger than the heap, and how a container gets OOM-killed.