Memory leaks and OOM kills

A garbage collector frees objects that cannot be reached. A leak in a garbage-collected language is memory that is still reachable, but that the program will never use again. The collector cannot help, because from its point of view the memory is live. Leaks are a common cause of memory that grows steadily until the process dies.

Common leak patterns

• Unbounded caches and maps. A map[string]*Session that gets an entry on login and is never pruned. Use a bounded LRU with a size limit or TTL.
• Listeners and callbacks. In Node, a listener added per request to a long-lived emitter and never removed. Node prints MaxListenersExceededWarning when one emitter has more than 10 listeners, which is often the only hint.
• Goroutine leaks. A goroutine blocked forever on a channel send with no receiver keeps its stack and everything it references alive. Thousands accumulate. Give goroutines a way to exit, usually a cancelled context.Context.
• ThreadLocals and static collections in Java servers, where pooled threads live forever.
• Holding a big object through a small one. Keeping a tiny slice alive keeps its whole backing array alive.

Walking through it: find the leak by diffing

The method is the same in every language. Take two heap snapshots some time apart, then compare them to see what grew.

$ curl -s http://localhost:6060/debug/pprof/heap > heap1.pb.gz
$ sleep 600
$ curl -s http://localhost:6060/debug/pprof/heap > heap2.pb.gz
$ go tool pprof -http=:8081 -diff_base heap1.pb.gz heap2.pb.gz
$ jcmd <pid> GC.heap_dump /tmp/heap.hprof
$ node --heapsnapshot-signal=SIGUSR2 server.js
$ kill -USR2 <pid>

The first three lines are Go, with net/http/pprof exposed. jcmd writes a JVM heap dump for Eclipse MAT or IntelliJ. The Node pair writes a snapshot on a signal, which you open in Chrome DevTools.

Look for types whose count grows without limit. Then follow the "retained by" or "dominator" path back to the root that holds them. It is usually one map or one list.

Why RSS is bigger than the heap

Each process sees its own virtual address space. Two numbers in top or ps confuse people:

• VSZ is the address space reserved. Go and the JVM reserve large ranges up front, so a small Go service can show many gigabytes of VSZ. That mostly does not matter.
• RSS is the physical RAM in use right now. It is what matters, and it is roughly what a container limit is measured against.

The classic surprise is that heap size is not process size. A JVM with -Xmx4g also uses memory for metaspace (class metadata), JIT-compiled code, thread stacks (1 MB per thread by default), direct byte buffers and GC bookkeeping. RSS can easily reach 4.5 to 5.5 GB. Put that JVM in a 4 GB container and it gets OOM-killed under load, even though "the heap is only 4 GB".

Walking through it: how a container gets OOM-killed

In Kubernetes and Docker, a Linux cgroup enforces the container's memory limit. When usage hits the limit and the kernel cannot reclaim enough, for example from page cache, the OOM killer sends SIGKILL.

Exit code 137  =  128 + 9 (SIGKILL)
kubectl describe pod  →  Reason: OOMKilled

There is no exception and no stack trace. The process just disappears. Check the exit code and the reason first.

Sizing heaps below the container limit

Leave room for non-heap memory:

• JVM: use -XX:MaxRAMPercentage=75 so the heap tracks the container limit. The default is only 25% of container memory, which wastes most of the pod.
• Node.js: --max-old-space-size=<MiB> caps V8's old generation. Set it below the limit, for example 1536 for a 2 GiB container. Buffers live outside the V8 heap, so leave room for them.
• Go: GOMEMLIMIT near 90% of the container limit, as covered in GC pauses and tuning.

Common mistakes

• Assuming a GC prevents all leaks. Reachable, unused objects stay alive.
• Sizing the heap to the container limit. Leave headroom for the non-heap memory.
• Restarting a leaking service instead of finding the leak. The restart hides the trend and delays the outage.

Check yourself

• A pod gets killed with exit code 137 and a 4 GB heap limit. What is the first thing to check? RSS against the container limit, because non-heap memory counts toward RSS.
• Two heap profiles, taken 10 minutes apart, show one map growing. What next? Follow the retained-by path from that map to the root holding it, then bound or prune the map.

Key takeaways

• A leak in a GC language is memory that is still reachable and no longer used. Collectors cannot free it.
• Diff heap profiles over time to find the type that keeps growing.
• RSS, not heap size, decides whether a container is killed. Exit code 137 means SIGKILL.
• Size heaps below container limits with MaxRAMPercentage, GOMEMLIMIT or --max-old-space-size.

Further learning

For the container side of OOM kills, see Kubernetes in production. For the GC knobs that set heap size, see GC pauses and tuning.