Guide

Craft of code

Production debugging

Correlate, stabilize, capture evidence. Do not thrash. Separate mitigation from the root-cause fix.

42 of 166 · 2 min read

Production debugging adds constraints: you can hurt customers, and you often cannot attach a debugger to a multi-tenant process. Correlate first. Do not thrash.

Warning

Restarting pods or "one more deploy" without a hypothesis can destroy evidence and deepen the outage. Prefer incident response roles: someone drives mitigation, someone investigates.

Walking through the checklist

  1. Scope — blast radius, start time, which versions/regions/flags.
  2. Correlate — deploys, config changes, traffic shifts, dependency status pages.
  3. Stabilize — roll back or disable the flag if you have a clear candidate; do not keep pushing speculative hotfixes.
  4. Capture — traces, logs for a sample of failing requests, core dumps if appropriate.
  5. Reproduce offline when you can; continue on a canary if you cannot.

Separate mitigation from root-cause fix in your notes. Rolling back a bad deploy stops the bleeding; understanding the null deref in the new parser prevents the next one. Teams that conflate the two either stay down too long hunting purity, or recover fast and never learn. Say both timescales out loud in the incident channel.

On-call rotations improve when every incident ends with "how we knew" artifacts: dashboards, runbook lines, and the failing test. That overlaps incident response and production ownership.

When you are stuck, restate the failure in one sentence with numbers: "p99 latency on /checkout went from 120 ms to 900 ms after deploy abc123, only in us-east-1." List killed hypotheses. Compare working vs broken. Rubber-duck or pair — fresh eyes catch "you are on the wrong cluster." Sleep, for non-outages: a fourth consecutive hour of diminishing returns loses to a fresh morning.

Ego is the enemy. The faster you say "I do not understand this yet" with a clear next experiment, the faster you get help that matters.

SDE2s are trusted to debug unfamiliar systems without panicking or thrashing. In interviews, walk a past bug with this structure. Interviewers listen for systematic narrowing, not heroics. Mention a time bisect or a metric correlation saved you from rewriting the wrong component.

Common mistakes

  • Speculative hotfixes. Each deploy destroys evidence.
  • Conflating rollback with root cause. Mitigation and learning are two timescales.
  • Heroics without numbers. Restate blast radius, start time, version, region.

Check yourself

  1. You have a clear bad deploy candidate. Stabilize how?

Roll back or disable the flag. Do not keep pushing speculative hotfixes.

  1. What two timescales do you say in the incident channel?

Mitigation now (rollback/flag) and the later root-cause fix (the null deref, the test, the runbook).

Key takeaways

  • Scope, correlate, stabilize, capture, then repro offline.
  • Do not destroy evidence with random restarts and extra deploys.
  • Mitigation and root-cause fix are both required; they are not the same step.

Further learning

Next: closing the loop.

Debugging

A scientific loop for finding bugs fast — reproduce, hypothesize, bisect, instrument — from local failures to production incidents without making things worse.

Incident response

Run an incident with clear roles, severity and communication. Mitigate first, debug second, then hand off to the postmortem. Start here for the map of incident pages.

Owning your code in production

On-call, alerts, runbooks, and the loop that makes you trustworthy — and better at building systems that survive users.