DNS outages and ndots

DNS is a single point of failure that teams often forget to monitor. Every service resolves names before it can reach anything else. When DNS fails, the failure looks like a dozen unrelated outages: timeouts in the API, errors in the database client, and broken links in the dashboard. The root cause is one lookup path that everything shares.

Two incidents show the two main ways this happens, and one configuration detail shows how the problem quietly multiplies.

Two famous outages

Dyn, October 2016. An attacker flooded Dyn, a large authoritative DNS provider, with traffic from a botnet of IoT devices. Dyn's servers couldn't answer, so customers' domains stopped resolving, including those of major websites. The sites were running fine. Users simply couldn't find them. The takeaway: a provider outage takes down every customer at once, so a single provider is a concentration risk. Many large companies now run authoritative DNS with two independent providers.

Facebook, October 2021. During maintenance, a command withdrew the routes to Facebook's own DNS servers from the internet. The authoritative servers were still running, but nobody could reach them. Resolvers couldn't find the company's domains, so services failed, and internal tools that depended on DNS also failed. Engineers lost access to the systems they needed to fix it. The takeaway: DNS depends on the same network it serves, and a recovery path should not require the thing that broke.

Both outages share one takeaway. Monitor DNS from outside your own network, and keep a way to reach your systems that doesn't depend on your own name servers.

The ndots setting in Kubernetes

Inside a cluster, names like payments are short. The resolver adds search domains before trying the name as-is. Kubernetes sets ndots:5 in each pod's resolver config, which means "treat a name as relative if it has fewer than 5 dots".

So a call to api.stripe.com, which has 2 dots, is treated as relative. The resolver tries each search suffix first:

api.stripe.com.default.svc.cluster.local.   → NXDOMAIN
api.stripe.com.svc.cluster.local.           → NXDOMAIN
api.stripe.com.cluster.local.               → NXDOMAIN
api.stripe.com.                             → answer

That is four lookups for one answer, and three of them are misses. Each miss adds latency and load on the cluster DNS, and misses are negative-cached only briefly. At high request rates, cluster DNS becomes a bottleneck.

Fixes, in order of preference:

1. Use fully qualified names with a trailing dot, such as api.stripe.com., so the resolver skips the search list.
2. Lower ndots for pods that call external services a lot, using dnsConfig in the pod spec.
3. Cache in the application where the runtime doesn't.

What to monitor

Track resolution latency and error rates, separately from application errors. Alert on NXDOMAIN and SERVFAIL spikes. Test resolution from at least two independent networks. When an outage starts, check DNS first, because a DNS failure can look like nearly any other problem.

Common mistakes

• Relying on one authoritative DNS provider, with no second provider for critical domains.
• Assuming a DNS outage means the servers are down. Routing or network changes can make them unreachable.
• Leaving ndots:5 in place for services that make heavy external calls, without measuring the extra lookups.

Check yourself

• Why does a Kubernetes pod make four lookups for api.stripe.com? Answer: with ndots:5, a two-dot name is tried against each search suffix before the bare name, and three of those are misses.
• Why did Facebook's engineers lose access during the 2021 outage? Answer: their internal tools depended on the same DNS and network that the bad routing change had removed from the internet.

Key takeaways

• DNS failures look like many unrelated outages, because every service depends on it.
• Use more than one authoritative provider, and monitor DNS from outside your network.
• In Kubernetes, ndots:5 multiplies lookups for external names. Use trailing dots or lower the value.

Further learning

Next, see DNS security for trusting and protecting the answers themselves.