Debugging CrashLoopBackOff

CrashLoopBackOff means a container starts, exits, and Kubernetes waits before trying again. The wait grows each time, from 10 seconds up to 5 minutes. It is a symptom, not a cause. This page is the order to check things in, so you find the cause quickly.

The big idea

The restart loop hides the crash. Each new container starts fresh, so the current logs often show only the first few seconds. The useful output is in the last crashed container, and in the events the cluster recorded.

Work from the outside in. Start with the pod's status and last state. Then read the logs from the crash itself.

Walking through the commands

These commands need a running cluster, so treat them as a sketch of the order to use:

kubectl get pods -n payments -o wide
kubectl describe pod checkout-7d9f8c-xk2m1 -n payments
kubectl logs checkout-7d9f8c-xk2m1 -n payments --previous
kubectl get events -n payments --sort-by='.lastTimestamp'

The order matters:

1. describe. Read Last State for the exit code and reason. Check for OOMKilled, failed probes and image pull errors.
2. logs --previous. The output of the container that just crashed.
3. Exit 137. The container was killed for using too much memory. Raise the memory limit or fix the leak.
4. Exit 1 or an app error. Look for bad config, a missing Secret, a failed migration or a wrong start command.
5. Never Ready. Check the readiness probe path and how long startup takes.

ImagePullBackOff is a different problem. It means the image could not be pulled, usually from registry auth, a wrong tag or a private image. Don't debug it like a crash.

Common mistakes

• Reading the current logs only. The crash is in --previous.
• Raising memory on an exit 1. Exit 1 is an app error, not memory pressure.
• Making liveness depend on the database. A database blip then restarts every pod.
• Confusing ImagePullBackOff with a crash loop. They need different fixes.

Check yourself

1. A container exits with code 137. What most likely happened? The kernel killed it for using too much memory. Raise the limit or fix the leak.
2. Why use kubectl logs --previous? The running container is a fresh restart. The crash output is in the previous one.

Key takeaways

• CrashLoopBackOff is a symptom. Find the exit code and the reason first.
• Read describe, then logs --previous, then events.
• Liveness should be shallow. Readiness can check dependencies.
• Keep migrations out of the startup path of the app.

Further learning

Next, see how the cluster runs in production, with resources, rollouts and GitOps, in Kubernetes in production.