Guide

Complex systems fail through interactions. A latent bug, a traffic spike, a missing alert and a slow rollback path can together cause an outage. Calling one of them the root cause hides the others. This page shows how to list contributing factors instead.

Concept

Before you start. Read the summary and timeline. You need the timeline to know which factors mattered and when.

The big idea

A 5 Whys exercise asks "why?" again and again, to get past the surface symptom. It is a useful prompt. It is a bad rule, because it tends to produce a single chain that ends at a person: "because Alice didn't check."

Use a graph instead of a ladder. Many factors branch and combine. Each one is a candidate for an action item.

Walking through a cache stampede

Product pages went down for 11 minutes. The team listed five factors.

  1. A hot cache key had the same expiry second on every replica.
  2. The expiry had no jitter, so every replica refreshed at once.
  3. The database connection pool was sized for steady traffic only.
  4. Mobile clients retried with no backoff, which amplified the load.
  5. The alert on database CPU fired at 95%, after the cliff.

Each factor got an action item: jitter on the TTL, pool headroom, retry budgets on clients, and an earlier saturation signal. Blaming "the engineer who set the TTL" would have fixed none of the amplification paths.

Grouping the factors

Sort factors into categories. The categories show which kinds of gap your reviews keep missing.

  • Code and config: the bug, the TTL setting.
  • Architecture: the shared pool, the lack of isolation.
  • Process: the missing canary, the unreviewed change.
  • Observability: the late alert, the missing dashboard.
  • People and capacity: on-call load, the team that was out.

What went well

Call out what worked. Fast mitigation, clear incident command and useful dashboards are habits to repeat on the next bad day. A postmortem that lists only failures teaches people to hide.

Common mistakes

  • Stopping at the first human in a 5 Whys chain. The chain ends where the fix is least useful.
  • Naming one root cause. Complex failures have several causes that combine.
  • Listing factors with no action items. A factor without a fix is only a story.

Check yourself

  1. Why group factors into categories? Answer: the categories show where gaps keep appearing, such as observability, so you can fix that area for good.
  2. In the cache stampede, why did the 95% CPU alert not help? Answer: it fired after the cliff, so it confirmed a problem the team was already seeing.

Key takeaways

  • A failure usually has several contributing factors that combine.
  • Use a graph, not a single chain that ends at a person.
  • Group factors by category, and name what went well.
  • Next, action items explains how to turn factors into finished work.

Further learning

The Google SRE book's example postmortem shows the template filled in, with several causes instead of one.

Example Postmortem (Google SRE book)This practical site reliability incident postmortem example outlines blameless analysis, action items and a detailed timeline for effective outage resolution.sre.google