Sharing postmortem learnings

A postmortem only teaches if people can find it. File each one in a searchable incident library, tagged by service and failure mode: deploy, dependency, capacity, data or security. Then add short reviews and readouts, so the lessons reach teams that were not on the incident.

Practices that scale

• Postmortem review. A 30 to 45 minute meeting with the authors, neighboring teams and a manager. The goal is to sharpen the factors and actions, not to re-argue blame.
• Cross-team readouts. Once a month, present two or three postmortems that teach a general lesson, such as retry storms or a bad regex.
• Templates and automation. Pull deploy IDs, alert history and graph screenshots into the document. Authors then spend their time on analysis, not on digging.
• Near-misses. Write a short note when you almost failed. Near-misses teach cheaper than outages.

Reading public postmortems

Mature vendors publish their postmortems. Read them for structure and candor, not for gossip.

Cloudflare, July 2, 2019. A managed WAF rule was deployed with a regex that backtracked catastrophically. It pinned CPU on HTTP-serving machines around the world. Traffic dropped sharply for roughly half an hour. The write-up does not stop at "bad regex." It explains how a global push skipped the usual staged rollout, and what the team changed afterward. The lesson: a config path that can take down the world needs the same rigor as a binary deploy.

GitLab, January 31, 2017. An accidental deletion met broken backups. The write-up is specific about what failed in the backup chain. It is a good example of publishing uncomfortable operational facts. For the recovery side, see disaster recovery.

GitHub and AWS write-ups often frame customer impact well. They separate what responders knew at minute 12 from what is known in hindsight. Copy that discipline.

When you read one, ask: would this template work on my team? Which action items would carry over? Where did detection lag?

Facilitating the review

Draft the timeline from machine data first. Then ask participants to fill the gaps. The author should not write the contributing factors alone late at night, because bias creeps in. A facilitator who was not the main operator can ask basic questions that experts skip, such as "why didn't the canary stop this?"

Keep the meeting blameless in how it runs, not only in its wording. Redirect "who approved this?" to "which gate was missing?" Thank people who escalate early, and people who say "I don't know yet."

Common mistakes

• Reviews that re-argue blame. Redirect every question toward gates and factors.
• Writing a long essay for a small incident. Match the depth to the severity.
• Writing the document and never checking the actions. Review open items in the weekly ops meeting.

Check yourself

1. Why tag postmortems by failure mode? Answer: so a team can search past deploy or capacity failures before it designs a similar change.
2. Why read public postmortems? Answer: for their structure, impact framing and candor, which you can copy into your own write-ups.

Key takeaways

• A searchable library is what lets the lessons spread.
• Short reviews and monthly readouts multiply the value of each outage.
• Read public write-ups for how they frame impact and cause.

Further learning

Cloudflare's blog keeps an index of its postmortems. The July 2019 write-up is the full account of the outage above.

Cloudflare blog: post-mortemsAll posts tagged with Post MortemCloudflare Blog Details of the Cloudflare outage on July 2, 2019Almost nine years ago, Cloudflare was a tiny company and I was a customer not an employee. Cloudflare had launched a month earlier and one day alerting told me that my little site, jgc.org, didn’t seem to have working…Cloudflare Blog