Networking
Debugging a slow endpoint
A hop-by-hop method for a p99 regression. Start with whether the median moved, pull traces of slow requests, and test one hypothesis per hop with the matching tool.
Your GET /products/:id endpoint's p99 jumped from 300 ms to 1.5 s yesterday, and the p50 barely moved. Where do you look? Start with the shape of the problem, then test one hypothesis per hop, using a tool built for that hop.
Concept
Before you start: p50 is the median response time. p99 is the time 99% of requests beat. A regression that moves only the p99 usually comes from something intermittent, while one that moves the p50 is systemic.
Walking through it: read the shape first
A problem that hits every request moves the median too. That points to a deploy, a config change, a dependency or capacity. A tail-only regression usually comes from something intermittent: GC pauses, lock contention, retries after timeouts, cache stampedes, or one bad host. Start with traces of slow requests specifically, and look at what they share.
Walking through it: one hypothesis per hop
Before reading the table, guess one cause per hop and the tool that would test it. Then check the table.
| Hop | Hypothesis | How to test |
|---|---|---|
| Client and network | Only users in one region or ISP are slow; packet loss on one path | Real-user data split by country and ISP; mtr from an affected region |
| DNS | Low TTL plus a slow or failing provider; ndots amplification in Kubernetes | dig timings, resolver metrics, CoreDNS latency dashboards |
| TLS | Session resumption broken after a load balancer change, so every connection does a full handshake | openssl s_client -reconnect, load balancer TLS metrics, the DevTools SSL phase |
| CDN | Cache hit ratio dropped, because a new Vary header or cookie made responses uncacheable | CDN analytics hit ratio, Age and X-Cache response headers |
| Load balancer | Uneven distribution, or retries after 502s from keep-alive mismatches | Per-target latency and 5xx metrics, access logs |
| App server | GC pauses, thread-pool or event-loop saturation, a slow new code path | Traces, a CPU profile or flame graph, GC logs, event-loop lag (profiling) |
| Downstream service | The recommendations service's own p99 rose, or calls became sequential | Traces: compare child-span durations and overlap |
| Cache | Hot keys expired together (a stampede), lowering the hit rate | Redis hit ratio, INFO stats, latency per command |
| Database | Plan changed after a statistics update; lock contention; table bloat | EXPLAIN ANALYZE, pg_stat_statements, pg_locks, slow query log |
| Host | Noisy neighbor, CPU throttling from container limits, disk saturation | Container throttling metrics, vmstat, iostat (Linux performance) |
Walking through it: a worked case
Say the traces of slow requests all show the same long span: a downstream call to the recommendations service, with its child spans now running one after another. The median did not move, because most requests skip that call. The fix is to make the calls concurrent, and to check the downstream service's own p99. Without traces, this would look like a random tail, and you would waste hours on GC or the database.
Common mistakes
- Starting with the code. Check the shape of the problem and the traces first.
- Restarting the service. A restart can clear the symptom and hide the trend, and the problem comes back.
- Testing many hypotheses at once. Change or check one thing per hop, so the result means something.
Check yourself
- The p99 jumped but the p50 did not. Where do you look first? At traces of slow requests only, to see whether one span or gaps between spans dominate.
- Every span in a slow trace looks normal, but there are large gaps between them. What might that be? Queueing, lock contention or GC pauses.
Key takeaways
- Check whether the median moved. A median move points to a systemic cause. A tail-only move points to something intermittent.
- Pull traces of slow requests and look at what they share.
- Test one hypothesis per hop, with the tool built for that hop.
- Tail latency compounds when a page fans out to many services, so a small per-service tail becomes a large page-level tail.
Further learning
Availability and latency explains why tail latency compounds across fan-out. For the tools that profile the app server hop, see performance and profiling.
Related
Measuring a request
How to read a browser waterfall, send Server-Timing headers, follow a request with distributed traces, and break down the 60 ms of server time for a product page.
Availability, latency and the tail
Nines, percentiles and the math of composing services. Why p99 matters more than the average, how fan-out amplifies the tail, and how to design for both uptime and speed.
Performance and profiling
How to find where time goes before you optimize. Percentiles, two predictive laws, sampling profilers, off-CPU waiting and honest benchmarks, with a worked investigation of a slow endpoint. Start here for the performance toolkit.