Craft of code
Bisecting bugs
Binary search the search space — code path, commits, input, config, cluster. git bisect is log2(N) builds.
When the search space is large, bisect: cut it in half, test the midpoint, repeat. This is the same idea as minimizing a repro, applied to any axis you can split.
git bisect is binary search over commits: mark a known good and known bad, test the midpoint, repeat. Roughly log2(N) steps for N commits — 1,000 commits ≈ 10 builds. Automate with git bisect run ./repro.sh when you have a scriptable check. Pair with git internals so you trust what bisect is checking out.
Walking through the axes
| Axis | Technique |
|---|---|
| Code path | Comment out / feature-flag half the pipeline |
| History | git bisect between last good and first bad commit |
| Input | Halve the request, dataset, or batch |
| Config | Diff env/flags with a known-good environment |
| Cluster | Compare healthy vs unhealthy instance (versions, CPU, neighbors) |
You need a repro that is cheap enough to run at each midpoint. If the check is "wait for the pager," bisect will not finish. Shrink until a script can say good or bad.
When you are stuck, the same idea is compare working vs broken: another region, previous binary, different customer tier, local vs prod config. List what you know is not true — killed hypotheses prevent loops.
Widen, then narrow again. Temporarily consider "it is DNS / certs / clock skew / disk full," check them quickly, discard. When the bug "cannot happen," check the plug: wrong cluster, stale build, feature flag off in that region, reading a replica lagging 30 seconds.
Keep these categories on a short mental list:
- Off-by-one and inclusive/exclusive ranges
- Time zones and DST (
TIMESTAMP WITHOUT TIME ZONElies) - Encoding (UTF-8 vs Latin-1, URL encoding twice)
- Caching (stale reads, cache stampedes — see caching)
- Races and lost updates
- Config drift between environments
- Idempotency failures on retries
- Floating point money (use integers/cents)
Common mistakes
- Linear "maybe this commit." Ten builds of bisect beat two hours of hunches on 1,000 commits.
- Bisect without a scriptable repro. You will mis-mark midpoints.
- Never listing killed hypotheses. You will retry the same story.
Check yourself
- About how many builds to bisect 1,000 commits?
About 10 — log2(1000).
- You cannot attach a debugger. Which axes still bisect?
Input, config, cluster (healthy vs unhealthy), and history if you can run the repro on old binaries.
Key takeaways
- Bisect any axis you can split: code, commits, input, config, cluster.
git bisect runneeds a cheap, scriptable repro.- Compare working vs broken, and write down killed hypotheses.
Further learning
A walkthrough of git bisect:
Next: reading the evidence.
Related
Debugging
A scientific loop for finding bugs fast — reproduce, hypothesize, bisect, instrument — from local failures to production incidents without making things worse.
Reproduce and minimize
The smallest steps, inputs and environment that make the bug happen reliably. Without a repro you are guessing.
Git internals and workflow
Git as a content-addressed object store plus a graph of commits and movable pointers, so merges, rebases, bisects and recoveries stop feeling risky, along with the team workflows that go with them.