Packets, layers and addresses

Network data travels as packets. Each packet is a small envelope with addresses on the outside and the real payload inside. Knowing how the envelopes nest, and how addresses are grouped, explains most of what you will debug on a network.

The layers you actually use

Textbooks teach a seven-layer model. Working engineers use four layers, because that is how the code is split:

Walking through it: a 1 KB write

When your service writes a 1 KB JSON body to a socket, the kernel builds nested headers around it:

[ Ethernet 14 B | IPv4 20 B | TCP 20–32 B | HTTP headers + 1 KB JSON | Ethernet FCS 4 B ]

Routers read only up to the IP header. They rebuild the Ethernet frame at every hop, because each hop has different MAC addresses. They also decrement the IP header's TTL (time to live) by one, so a looping packet eventually dies. Only the two endpoints look at TCP and above. Middleboxes such as NATs and firewalls break this rule, which is why they cause so many surprises.

The MTU (maximum transmission unit) of most Ethernet and cloud networks is 1,500 bytes. Subtract 20 bytes of IPv4 header and 20 of TCP header, and you get the MSS (maximum segment size) of 1,460 bytes. A 100 KB response is therefore about 69 full-size packets.

IP addresses and CIDR

An IPv4 address is a 32-bit number written as four bytes, such as 10.0.3.17. CIDR notation marks how many leading bits are fixed. In 10.0.0.0/16, the first 16 bits are the network, so the block holds 2^16 = 65,536 addresses. A /n block always holds 2^(32 − n) addresses.

Not every address is usable. The first is the network address and the last is the broadcast address. AWS reserves five per subnet, leaving 251 of a /24. This matters for Kubernetes, where each pod may use a VPC IP. A /24 per availability zone runs out at around 250 pods, which is a common outage.

Private ranges (RFC 1918) are never routed on the public internet: 10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16. Your laptop, your VPC and your pods almost certainly use one of them.

NAT

NAT (network address translation) lets private hosts reach the internet. A NAT gateway rewrites the source address and port of outgoing packets to its own public IP and a free port. It remembers the mapping, so replies can be translated back.

Two failures follow. First, one public IP can hold only about 64,000 concurrent connections to a single destination IP and port, because that is how many source ports exist. Services that open thousands of short connections to one API hit port exhaustion. Second, NAT mappings expire when idle, often after a few minutes. A long-lived idle connection can then die silently. That is why database drivers and gRPC clients send keepalives.

IPv6 uses 128-bit addresses, such as 2001:db8::1. Its header is 40 bytes, so the MSS on a 1,500 MTU drops to 1,440. Most production systems run dual-stack: DNS returns both A (IPv4) and AAAA (IPv6) records, and clients race them with the "Happy Eyeballs" algorithm.

Common mistakes

• Sizing a subnet for nodes and forgetting pods. Each pod can consume an address.
• Assuming NAT is free. It has port limits and idle timeouts.
• Confusing MTU and MSS. MTU is the link limit. MSS is the TCP payload after headers.

Check yourself

• How many addresses does a /26 hold, and how many are usable in a classic subnet? 2^6 = 64 addresses. Two are reserved, leaving 62 hosts.
• Why does a NAT gateway run out of connections to one API before it runs out of memory? Each connection needs a source port, and one public IP has only about 64,000 usable ports per destination.

Key takeaways

• Packets nest like envelopes: link, IP, transport, application. A TCP segment carries about 1,460 bytes of payload on a 1,500-byte MTU.
• A /n block holds 2^(32 − n) addresses. Private ranges and NAT let most networks share public addresses.
• NAT brings port exhaustion and idle timeouts, which cause mysterious connection failures.

Further learning

Next, see how TCP turns these packets into a reliable stream, in TCP handshake and reliability.

For how cloud VPCs and subnets are built on these ideas, see cloud networking.