Congestion control and BDP

Flow control stops a sender from overrunning the receiver. The real bottleneck is often somewhere in the middle, on a congested peering link, a home Wi-Fi router or a cell tower. Nobody tells the sender how much capacity exists, so TCP has to find out. It keeps a second limit, the congestion window (cwnd). The amount of data in flight is min(cwnd, rwnd).

Slow start

Slow start is how a connection begins. Modern Linux starts with an initial window of 10 segments (RFC 6928), about 10 × 1,460 = 14,600 bytes. Every ACK grows cwnd by one segment, which doubles the window each round trip. Despite the name, that growth is exponential.

Growth continues until a loss happens or cwnd crosses the slow-start threshold. After that, growth becomes gentler, in a phase called congestion avoidance.

Walking through it: sending 100 KB on a fresh connection

Say the server needs to send a 100 KB response. The RTT is 100 ms and the link is 100 Mbps.

Round trip 1: TCP handshake                            (0 bytes of data)
Round trip 2: request goes out, server sends 14.6 KB   (total 14.6 KB)
Round trip 3: cwnd = 20 segments → 29.2 KB             (total 43.8 KB)
Round trip 4: cwnd = 40 segments → 58.4 KB             (total 102.2 KB)  done

Time ≈ 4 RTTs × 100 ms = 400 ms
Pure transmission time at 100 Mbps: 100 KB × 8 = 800,000 bits / 100,000,000 b/s = 8 ms

The link could have moved the bytes in 8 ms. Latency cost 392 ms of extra time. This is why web performance advice says to keep the critical first response under about 14 KB, which fits in the first flight. It is also why a reused connection is so much faster. Its cwnd has already grown.

CUBIC vs BBR

How the window reacts to trouble is set by the congestion control algorithm. You can choose it per host.

• CUBIC, the Linux default since 2006, is loss-based. It grows the window along a cubic curve and cuts it by 30% when a packet is lost. On paths with large buffers, it keeps pushing until the buffers overflow. That inflates latency, a problem called bufferbloat. On lossy wireless links it mistakes random loss for congestion and backs off for no reason.
• BBR, from Google in 2016, is model-based. It estimates the bottleneck bandwidth and the minimum RTT, and paces packets to match bandwidth × minimum RTT. It does not fill buffers. Google reported large throughput gains on YouTube and lower latency on lossy paths. BBR can be unfair to CUBIC flows that share a bottleneck, which is part of why BBRv2 and BBRv3 exist.

Switching is one sysctl (net.ipv4.tcp_congestion_control=bbr). Many CDNs and video services run BBR at their edge. Inside one data center the choice rarely matters. For users on long or lossy paths, it can.

Bandwidth-delay product

A long, fat pipe holds a lot of data in transit. The bandwidth-delay product (BDP) is how many unacknowledged bytes must be in flight to keep the pipe full:

BDP = bandwidth × RTT

Example: replicate a database from us-east-1 to eu-west-1
  bandwidth = 1 Gbps, RTT = 80 ms
  BDP = 1,000,000,000 b/s × 0.080 s = 80,000,000 bits = 10 MB

If either window is smaller than 10 MB, the sender sits idle waiting for ACKs, and the full gigabit is not used. Throughput is capped at window ÷ RTT:

With a 64 KB window (no window scaling):
  65,535 B / 0.080 s = 819 KB/s ≈ 6.5 Mbps   (0.65% of the link)

With a 4 MB window:
  4,194,304 B / 0.080 s ≈ 52 MB/s ≈ 420 Mbps

This explains why a cross-region scp or replication stream can crawl on a fast link. The fix is buffer tuning (net.ipv4.tcp_rmem and tcp_wmem) or parallel streams, not more bandwidth.

For ordinary web traffic, the lesson is that round trips dominate small requests. A typical API response is 1 to 50 KB. On a 100 Mbps link, its transmission takes under 4 ms, but each round trip costs 20 to 200 ms depending on distance. Cutting round trips beats adding bandwidth almost every time. Light in fiber covers about 200,000 km per second, so New York to Sydney, about 16,000 km, takes at least about 160 ms round trip.

Common mistakes

• Buying bandwidth for a latency problem. Small responses are limited by round trips.
• Using default buffers on long-distance transfers. A 64 KB window caps a gigabit link at under 1%.
• Assuming the window is always large. A new connection starts at 10 segments.

Check yourself

• Why does the 100 KB transfer take 400 ms when the link needs only 8 ms? Slow start needs four round trips to grow the window, and each one costs 100 ms.
• A 1 Gbps link with an 80 ms RTT is limited to 6.5 Mbps by a 64 KB window. Why? Throughput equals window divided by RTT, and 65,535 bytes per 0.08 s is about 6.5 Mbps.

Key takeaways

• A new connection starts with a 10-segment window and doubles it each round trip, so it needs several round trips to reach full speed.
• CUBIC grows until it loses packets. BBR paces to the measured bandwidth and RTT.
• The bandwidth-delay product tells you how much data must be in flight. Throughput is capped at window divided by RTT.
• For small API responses, remove round trips before buying bandwidth.

Further learning

Next, see what TCP's ordering costs in practice, and when UDP is the right choice, in TCP latency traps and UDP.

Wireshark's "Time Sequence" graph shows slow start as a staircase in a real capture, covered in network tools.