Mitigate before root cause

During an incident, the first goal is to stop users from failing. The explanation can come later. Use reversible, practiced actions first, and dig for the root cause once the error rate has fallen.

The big idea

Debugging feeds the need to understand. Mitigation feeds the users. An incident is a fire, and the first move is to shut the gas valve, even before you know which pipe leaked.

The fire analogy has a limit. Some valves are one-way. A database migration that cannot be reversed is not something you can simply shut off. For those, use a flag, failover or load shedding instead, and practice those paths before you need them.

The mitigation menu

Walking through a choice

In the connection pool incident, the team had two options. The first was to scale up the payments pods. The second was to roll back the deploy that added a heavier query.

Scaling looks like the obvious lever. But each new pod opens its own connections to the same pool, so the pool stays exhausted. The commander chose rollback, because it removes the load that caused the exhaustion. Raising the pool size was the fallback only if the rollback was blocked.

Hypothesis discipline

A hypothesis is a claim you can test, such as "payments pool exhaustion from the 13:55 deploy." Keep the discipline:

1. State it out loud.
2. Give it one owner.
3. Time-box it to 10 to 15 minutes.
4. Then either mitigate with it, or discard it.

The scribe writes each hypothesis down, so nobody retries a dead end.

Risky mitigations need an explicit go

Some levers carry real risk. Failover that can lose recent writes, dropping traffic for a whole region, or running a hot-fix SQL statement against the primary all need a go from the IC. Before the go, check three things: the worst case, how to undo it, and who watches the graphs for two minutes after. The operations lead proposes, and the IC decides.

If two mitigations conflict, pick one, time-box it, and then evaluate. Parallel, uncoordinated changes make the timeline unreadable.

Common mistakes

• Rolling back with no clean artifact. Know the last good version before you need it.
• Scaling a service whose pool is exhausted. More pods can make it worse.
• Retrying a dead end. The scribe's log prevents that.
• Assuming a one-way migration can be rolled back. Check first, and prepare flags.

Check yourself

1. Why choose rollback over scaling when a connection pool is exhausted? More pods open more connections to the same pool, so scaling does not free it. Rollback removes the cause.
2. How long should you spend on a single hypothesis before deciding? About 10 to 15 minutes, then mitigate with it or discard it.

Key takeaways

• Restore service first, explain second.
• Prefer reversible levers: rollback, flags, failover and load shedding.
• Time-box each hypothesis and log it.
• Risky changes need the commander's explicit go.

Further learning

Next, see how to keep customers and leadership informed while you work in communication and handoffs.