Guide

Craft of code

Reading debug evidence

Stacks, logs, metrics and traces. The exception escaped somewhere; the mistake may be a frame earlier.

41 of 166 · 2 min read

Stack traces point to where the exception escaped, not always where the mistake began — walk frames until the values stop making sense.

Logs need correlation IDs. Searching error in a 10 GB file without a request id is archaeology. Add targeted logs that print the hypothesis variables (user id, flag value, branch taken), then remove or demote them after the fix.

Metrics and traces answer "when" and "where in the graph." A latency spike starting at deploy time is a different beast from a gradual memory climb. Use profiling when the bug is "slow," not "wrong."

Decide what variable you need to observe, then pick the tool that shows it fastest. Jumping into a debugger "to look around" is wandering.

Walking through tools

Worth fluency:

  • Interactive debuggers (breakpoints, conditional breaks, watch expressions)
  • strace / dtruss / Process Monitor for unexpected syscalls and files
  • Browser and proxy inspectors for HTTP
  • Distributed tracing (OpenTelemetry) for multi-service paths
  • Memory and CPU profilers for leaks and hot loops
  • Race detectors (go test -race, TSan) for concurrency bugs

Build a personal cheat sheet for your stack: how to capture a heap profile, how to enable verbose gRPC logging, where sampling traces live, how to decrypt a local JWT. Five minutes of setup documentation saves an hour during an incident.

Example

Intermittent Unauthorized on a service-to-service call. Hypothesis 1: token expired — logs show 5-minute-old tokens. Hypothesis 2: clock skew — compare NTP on both sides, fine. Hypothesis 3: audience claim mismatch after a config reload — diff the JWT aud field on failing calls; mismatch confirmed. Fix: align config; add a metric on auth reject reasons so the next time is a graph, not a mystery.

Trade-off

Permanent high-cardinality logs are expensive and noisy. Prefer temporary targeted logs or dynamic debug levels during the hunt, then keep only what helps operate the system afterward.

"Slow" uses the same loop with different evidence: CPU and allocation profiles, EXPLAIN, lock waits, flame charts. Hypotheses sound like "N+1 queries on line items" or "the regex is backtracking." Gut-driven caching adds complexity for a millisecond you invented.

Common mistakes

  • Trusting the top frame. Walk until values stop making sense.
  • Grep error with no request id. Correlation first.
  • Debugger tourism with no variable you intend to observe.

Check yourself

  1. A stack shows a null in a helper. Where might the mistake actually be?

A frame earlier — where a bad value was produced and passed down. Walk until values stop making sense.

  1. After the JWT aud mismatch, what evidence should you leave behind?

A metric on auth reject reasons so the next time is a graph.

Key takeaways

  • Walk stacks; correlate logs; use metrics and traces for when and where.
  • Pick the tool that shows the hypothesis variable fastest.
  • Keep surgical instrumentation; drop noisy permanent logs.

Further learning

Next: production debugging.

Debugging

A scientific loop for finding bugs fast — reproduce, hypothesize, bisect, instrument — from local failures to production incidents without making things worse.

The scientific debugging loop

Observe a concrete failure, form a killable hypothesis, change one variable, narrow, repeat. Write it down.

Performance and profiling

How to find where time goes before you optimize. Percentiles, two predictive laws, sampling profilers, off-CPU waiting and honest benchmarks, with a worked investigation of a slow endpoint. Start here for the performance toolkit.