Craft of code
Debugging
A scientific loop for finding bugs fast — reproduce, hypothesize, bisect, instrument — from local failures to production incidents without making things worse.
Debugging is not staring at code until inspiration hits. It is a loop: observe a concrete failure, form a falsifiable hypothesis, run an experiment that could disprove it, narrow the search space, repeat. The goal is the root cause, not the first symptom that went red.
SDE2s are trusted to debug unfamiliar systems without panicking or thrashing. Narrate the loop: current hypothesis, next experiment, what would falsify it. In interviews, walk a past bug with that structure. Interviewers listen for systematic narrowing, not heroics.
Write hypotheses as sentences you can kill. Change one variable per experiment. Keep an audit trail after hour two. If you did not fix the original failure, it is not fixed — do not declare victory on a different symptom. git bisect over 1,000 commits is about 10 builds when the repro is scriptable.
Concepts in this section
Read them in order.
| Step | Concept | What it answers |
|---|---|---|
| 1 | The scientific loop | What do you write down before you change anything? |
| 2 | Reproduce and minimize | How small can the failing case get? |
| 3 | Bisecting bugs | How do you cut code, commits, input and config in half? |
| 4 | Reading the evidence | What do stacks, logs, metrics and traces actually say? |
| 5 | Production debugging | How do you mitigate without destroying evidence? |
| 6 | Closing the loop | What test, note and metric stop this from returning? |
A 4 MB upload
API returns 500 for "some" uploads. Binary-search file size: failures start at 4,194,304 bytes. Hypothesis shifts from "virus scan" to "body size limit." The proxy max body is 4 MB. Root cause in minutes, not a day of MIME types. That is minimize-the-input, then one config check — the loop in miniature.
In production, add: scope the blast radius, correlate with deploys and flags, stabilize (rollback or flag off), capture traces, then repro offline. Restarting pods without a hypothesis can destroy evidence. Separate mitigation from the later root-cause fix.
Common mistakes
- Theorizing without a repro. Especially on races.
- Changing three things at once. You cannot tell what mattered.
- Thrashing production. Speculative deploys deepen the outage.
- Fix without a test. The bug returns after the next refactor.
- Grep
errorwith no request id. Correlate first.
Key takeaways
- Observe → hypothesize → experiment → narrow; write hypotheses down.
- Invest in a reliable, minimized repro; bisect across code, commits, inputs and config.
- In production: correlate, mitigate safely, preserve evidence.
- Ship the fix with a regression test and a short write-up.
- When stuck, restate facts, kill hypotheses, compare working vs broken, then ask for help with an audit trail.
Further learning
A walkthrough of git bisect:
Related
Performance and profiling
How to find where time goes before you optimize. Percentiles, two predictive laws, sampling profilers, off-CPU waiting and honest benchmarks, with a worked investigation of a slow endpoint. Start here for the performance toolkit.
Incident response
Run an incident with clear roles, severity and communication. Mitigate first, debug second, then hand off to the postmortem. Start here for the map of incident pages.
Git internals and workflow
Git as a content-addressed object store plus a graph of commits and movable pointers, so merges, rebases, bisects and recoveries stop feeling risky, along with the team workflows that go with them.