Latency budget and round trips

For a distant user, the biggest cost in a request is usually the number of round trips, not the server's work. A round trip is the time for a packet to go out and come back. Count them on the critical path, multiply by the round-trip time, and you have most of the answer.

The setup

A user in Mumbai loads a product page. The origin is in AWS us-east-1, in Northern Virginia, about 13,000 km away. The theoretical minimum round trip is:

2 × 13,000 km / 200,000 km/s = 0.13 s = 130 ms

Real paths are not straight lines and pass through many routers, so the measured RTT is typically around 200 ms. We use 200 ms. The HTML is dynamic, the server needs 60 ms to produce it, and it is 30 KB compressed.

Walking through it: scenario A, straight to the origin

DNS (resolver in Mumbai, miss → authoritative)          30 ms
TCP handshake                         1 RTT            200 ms
TLS 1.3 handshake                     1 RTT            200 ms
Request → first byte                  1 RTT + 60 ms    260 ms
Rest of 30 KB HTML (slow start: first flight ~14 KB,
   needs one more round trip)         1 RTT            200 ms
                                                       ------
HTML fully received                                    890 ms

That is nearly 900 ms before the browser has the HTML, and it has not fetched a single stylesheet yet. Only 60 ms of it is your code. The other 830 ms is round trips. Four of them cross the ocean.

Scenario B: through a CDN point of presence in Mumbai

Now add a CDN. Its PoP is in Mumbai, 10 ms from the user. The CDN keeps a pool of warm, long-lived connections from its edge to the origin, with large congestion windows and no handshakes to repeat.

DNS (CDN's anycast DNS, nearby)                         20 ms
TCP handshake to edge                 1 RTT (10 ms)     10 ms
TLS 1.3 handshake at edge             1 RTT (10 ms)     10 ms
Request → edge                        ½ RTT              5 ms
Edge → origin on warm connection      1 RTT + 60 ms    260 ms
Edge → user, first byte               ½ RTT              5 ms
Rest of HTML (edge-to-user slow start) 1 RTT            10 ms
                                                       ------
HTML fully received                                    320 ms

Where the 320 ms goes: 200 ms is the unavoidable trip to Virginia and back, 60 ms is the server, and about 60 ms is local DNS, handshakes and slow start at a 10 ms round trip. The CDN caches nothing here, yet it saves 570 ms. It does that by ending TCP and TLS close to the user and reusing warm connections to the origin.

Scenario C: CDN cache hit

For a cacheable page, or the CSS, JavaScript and images that follow, the edge answers by itself:

DNS 20 + TCP 10 + TLS 10 + request/response 10 + edge lookup 2 + rest of body 10
= 62 ms

That is 890 ms down to 62 ms, a 14× improvement, with no application changes. The first move for any global latency problem is to make content cacheable and serve it from the edge. For dynamic requests, the 200 ms that remains can only be cut by moving data and compute closer to users. See multi-region.

Common mistakes

• Profiling the handler first. For distant users the handler is often the smallest term in the budget.
• Forgetting the slow-start round trip. A response bigger than the first flight costs at least one more trip.
• Assuming a CDN only helps cache hits. Misses still benefit from local handshakes and warm origin connections.

Check yourself

• Why does the direct cold request need four round trips across the ocean? DNS, TCP, TLS and the request each wait on the connection to Virginia, plus one more for the rest of the HTML.
• Why does a CDN miss still save about 570 ms? The handshakes happen at a 10 ms RTT, and the edge reuses warm connections to the origin.

Key takeaways

• Latency is round trips multiplied by RTT. Count the round trips first.
• At 200 ms RTT, a cold HTTPS request spends about 830 ms on round trips before your server's 60 ms counts.
• A CDN cuts handshakes and slow start even when content is not cached. On a hit, it removes the ocean from the path.

Further learning

Next, see how reusing connections removes most of these round trips on the second request, in connection reuse.