Debugging connectivity

When one service cannot reach another, the fastest route is a checklist, not a guess. Go from the bottom up through the layers, and write down what each one told you. Most "the database is down" incidents end at one of the first three steps.

The checklist

1. DNS. Does the name resolve to the address you expect? Use dig. A private zone and a public zone with the same name is a classic cause.
2. Route. Does the subnet have a route to that address? A wrong route-table association after a Terraform change is common.
3. Firewall on both sides. Check the security group or firewall on the destination for inbound, and on the source if its egress is restricted.
4. Placement. Is the instance in a private subnet with no NAT, when it needs the internet?
5. NACL ephemeral ports. A locked-down NACL can drop the replies.
6. Overlapping ranges. Overlaps after peering make routes do something other than what you expect.

Reading the symptom

The error message says where to look.

• Timeout: the packets were dropped somewhere, by a filter, a missing route or a host with no listener.
• Connection refused: the packets reached a host, and its port is closed. The route and the filters worked.

Walking through an incident

An app times out connecting to rds.prod.internal:5432. The steps:

1. dig returns 10.16.140.10, so DNS is fine.
2. The reachability analyser reports that sg-db allows 5432 from sg-app. The rule looks right.
3. The connection attempt used an ENI that belongs to an old instance, still tagged sg-legacy after a cutover.
4. VPC flow logs show REJECT on the database's ENI.

The fix is to correct the security group membership. The database was never down.

Tools

• Reachability Analyzer (AWS) and Connectivity Test (GCP) check the path between two network interfaces.
• VPC Flow Logs record accepts and rejects. Turn them on before an incident, because they are the proof of which side denied the traffic.
• On the host: nc -vz, curl -v, ip route get and ss are the same tools as in Linux for engineers.

Common mistakes

• Declaring a service down from a timeout. A timeout usually means a filter, a route or a stale group, not a crashed service.
• Checking only one side's firewall. A rule on the destination does not help if the source's egress is restricted.
• Turning on flow logs during the incident. Enable them in advance.

Check yourself

1. A client gets "connection refused" rather than a timeout. What does that tell you? Answer: The packets reached a host, so the route and filters worked. The port on that host is closed or has no listener.
2. Why enable VPC flow logs before an incident? Answer: They record accepts and rejects, which is the evidence of which side denied the traffic.

Key takeaways

• Debug bottom-up: DNS, route, firewall on both sides, placement, NACL ports, overlaps.
• A timeout points at the path. A refusal points at the listener.
• Reachability Analyzer and flow logs turn guesses into evidence.

Further learning

The host-side commands in this page work the same way as the ones in Linux for engineers. The hub for the whole topic is cloud networking and VPCs.