Headroom for failure

Capacity is not "enough for the happy-path peak". Systems fail at peak, deploys consume spare capacity, and retries multiply load. Headroom covers those cases. A practical rule is this. After one zone fails during peak, the surviving capacity should stay at or below about 70 to 80% utilization on the limiting resource.

Four budgets

Zone or node loss (N+1). Provision for peak plus one failure domain. Suppose you need 10 pods' worth of work at peak, and one zone holds a third of the pods.

Survivors must carry 10 pods of work at 75% utilization:
  10 / 0.75 = 13.3, so 14 surviving pods
Two zones hold the 14 survivors, so the lost zone held 7 as well:
  total = 14 x 3 / 2 = 21 pods

The factor 3 / 2 is the one from the zones page. It is N × n / (n − 1) with three zones.

Deploys and rollouts. A rolling update needs surge capacity, such as Kubernetes maxSurge. Canaries briefly run two versions at once. Reindexing jobs use capacity too.

Autoscaling lag. Autoscalers react in minutes. Spikes can arrive in seconds. Keep a baseline that absorbs the fast part of a spike, and let autoscaling handle the slow part.

Retry amplification. Clients retry failed requests, so a small outage can multiply the load. A 10% outage can become a 2 to 3 times load storm. A capacity plan without retry budgets is incomplete. The resilience page covers retry limits in detail.

Common mistakes

• Provisioning for an average day. Losing a zone at peak is then an outage.
• Forgetting deploy surge. A two-pod service with maxUnavailable: 50% loses half its capacity on each step.
• Planning without retry limits. Clients then multiply an outage into a bigger one.

Check yourself

1. Why does a rolling deploy need spare capacity? Answer: New pods start before old ones stop, so the fleet briefly needs room for both.
2. How can a 10% outage become a 2 to 3 times load storm? Answer: Clients retry failed requests, which adds to the normal traffic on the struggling service.

Key takeaways

• Size for the survivors. Keep them under about 70 to 80% after losing a zone at peak.
• Include surge capacity for deploys and canaries.
• Keep a baseline that absorbs spikes faster than autoscaling reacts.
• Cap retries, or your headroom disappears during an incident.

Further learning

Next, see the full planning exercise on one realistic launch: worked example: a 5× launch.