Observability and reliability
Contributing factors, not a single root cause
Complex failures come from several causes that combine. List factors by category, and do not stop at the first person, because that fixes nothing.
Complex systems fail through interactions. A latent bug, a traffic spike, a missing alert and a slow rollback path can together cause an outage. Calling one of them the root cause hides the others. This page shows how to list contributing factors instead.
Concept
Before you start. Read the summary and timeline. You need the timeline to know which factors mattered and when.
The big idea
A 5 Whys exercise asks "why?" again and again, to get past the surface symptom. It is a useful prompt. It is a bad rule, because it tends to produce a single chain that ends at a person: "because Alice didn't check."
Use a graph instead of a ladder. Many factors branch and combine. Each one is a candidate for an action item.
Walking through a cache stampede
Product pages went down for 11 minutes. The team listed five factors.
- A hot cache key had the same expiry second on every replica.
- The expiry had no jitter, so every replica refreshed at once.
- The database connection pool was sized for steady traffic only.
- Mobile clients retried with no backoff, which amplified the load.
- The alert on database CPU fired at 95%, after the cliff.
Each factor got an action item: jitter on the TTL, pool headroom, retry budgets on clients, and an earlier saturation signal. Blaming "the engineer who set the TTL" would have fixed none of the amplification paths.
Grouping the factors
Sort factors into categories. The categories show which kinds of gap your reviews keep missing.
- Code and config: the bug, the TTL setting.
- Architecture: the shared pool, the lack of isolation.
- Process: the missing canary, the unreviewed change.
- Observability: the late alert, the missing dashboard.
- People and capacity: on-call load, the team that was out.
What went well
Call out what worked. Fast mitigation, clear incident command and useful dashboards are habits to repeat on the next bad day. A postmortem that lists only failures teaches people to hide.
Common mistakes
- Stopping at the first human in a 5 Whys chain. The chain ends where the fix is least useful.
- Naming one root cause. Complex failures have several causes that combine.
- Listing factors with no action items. A factor without a fix is only a story.
Check yourself
- Why group factors into categories? Answer: the categories show where gaps keep appearing, such as observability, so you can fix that area for good.
- In the cache stampede, why did the 95% CPU alert not help? Answer: it fired after the cliff, so it confirmed a problem the team was already seeing.
Key takeaways
- A failure usually has several contributing factors that combine.
- Use a graph, not a single chain that ends at a person.
- Group factors by category, and name what went well.
- Next, action items explains how to turn factors into finished work.
Further learning
The Google SRE book's example postmortem shows the template filled in, with several causes instead of one.
Example Postmortem (Google SRE book)This practical site reliability incident postmortem example outlines blameless analysis, action items and a detailed timeline for effective outage resolution.sre.googleRelated
Blameless postmortems
Learning from failure with timelines, contributing factors and action items that actually get done — without turning people into the root cause.
Writing the summary and timeline
Start with user impact, then build the timeline from machine data. Record the silent window between failure and detection, because it is often the most important line.
Action items that get finished
Good action items are specific, owned, prioritized and dated. Tracking them is what turns a postmortem into an actual fix.