Disruption budgets and spreading replicas

Some pod deaths are planned: a node drain, a cluster upgrade, or the autoscaler consolidating nodes. A PodDisruptionBudget (PDB) caps how many of a service's pods can be taken down by those voluntary disruptions at the same time. Spreading replicas across nodes and zones does the rest. Together they keep planned maintenance from becoming an outage.

The big idea

Think of a team on a shift rota. You would not let every person take the same week off, or the shop closes. The PDB is the rule "at most one of us away at a time". Spreading the team across floors means one fire does not empty the building.

Walking through it

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: checkout
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: checkout

With three replicas and maxUnavailable: 1, a drain can evict one pod, and it must wait until that pod is replaced before evicting the next. Without a PDB, a drain can evict every replica of the Deployment at once.

A PDB covers voluntary disruptions only. It does not protect you when a node crashes. It also interacts badly with small replica counts. With minAvailable: 2 and only two replicas, nothing can be disrupted, and a node upgrade can wait forever. For small services, prefer maxUnavailable, and run at least three replicas for anything customer-facing.

Spreading keeps one failed node from taking out the whole service:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: checkout

maxSkew: 1 means no zone can have more than one pod more than any other zone. Pod anti-affinity does the same across nodes. Use the soft form, preferredDuringSchedulingIgnoredDuringExecution. The hard form on a small cluster can leave pods Pending.

Common mistakes

• No PDB on a customer-facing service. A drain can remove every replica at once.
• minAvailable equal to the replica count on a small deployment. No disruption is allowed, so upgrades stall.
• Hard anti-affinity on a small cluster. Pods stay Pending when there are not enough nodes.

Check yourself

1. A service has three replicas and a PDB with maxUnavailable: 1. A drain starts. How many pods can be evicted at once? One. The next eviction waits until the replacement is up.
2. Does a PDB protect against a node crash? No. A PDB covers voluntary disruptions. A crash is involuntary.

Key takeaways

• A PDB caps voluntary disruptions. Use maxUnavailable for small services.
• Run at least three replicas for customer-facing services.
• Spread replicas with topology spread and soft anti-affinity.

Further learning

The deploy side of these rules is in rollouts and graceful termination. The platform basics are in Kubernetes in production.

Kubernetes in production