Closing the debug loop

A fix without a regression test is a temporary fix. Add the minimized repro as a test. Write a short note (PR description or incident timeline): what we saw, what we hypothesized, what was true, how we prevented recurrence. That note is gold for the next person — including you in six months.

After you fix something non-obvious, invest five more minutes in future you: a dashboard panel, a runbook bullet ("if auth rejects spike, check aud config"), or a metric label that would have made today's hypothesis #3 appear first. Debugging skill is partly personal; institutionalizing the lesson is how on-call gets easier for everyone.

Walking through stuckness

Every engineer hits a wall. Productive stuckness has a checklist:

1. Restate the failure in one sentence with numbers: "p99 latency on /checkout went from 120 ms to 900 ms after deploy abc123, only in us-east-1."
2. List what you know is not true — killed hypotheses prevent loops.
3. Compare working vs broken — another region, previous binary, different customer tier, local vs prod config.
4. Rubber-duck or pair — explaining the audit trail often reveals the gap.
5. Widen, then narrow again — DNS / certs / clock skew / disk full, check quickly, discard.
6. Sleep — for non-outages, a fresh morning beats a fourth hour of diminishing returns.

When you help someone, do not steal the keyboard immediately. Ask what they have tried, what the current hypothesis is, and what experiment is next. Afterward, push for the regression test and a short write-up in the ticket — that is how the team's collective debugging skill compounds.

Document races you found; concurrency bugs that are "fixed" without a regression test tend to return after the next innocent refactor.

Separate mitigation from root-cause in the write-up, as in production debugging: the rollback stopped the bleeding; the parser test and the aud metric prevent the encore.

Common mistakes

• Fix, no test. It will return after the next refactor.
• No write-up. Six months later you are the next person, and you remember nothing.
• Stealing the keyboard when mentoring. Pair on the method so the skill compounds.

Check yourself

1. What belongs in the short note besides "we fixed it"?

What we saw, what we hypothesized, what was true, and how we prevented recurrence.

2. You found an aud mismatch. What five-minute investment helps the next on-call?

A metric on auth reject reasons, or a runbook bullet: if auth rejects spike, check aud config.

Key takeaways

• Ship the fix with a regression test and a short write-up.
• Institutionalize the lesson: dashboard, runbook, metric.
• When stuck, restate facts, kill hypotheses, compare working vs broken, then ask for help with an audit trail.

Further learning

A debugging manifesto (Julia Evans)A debugging manifestoJulia Evans

Back to the map: debugging.