Rollout strategies

There are four common ways to move from one version of a service to the next: rolling, blue-green, canary and shadow. The choice is about how much risk you accept, how much spare capacity you can pay for, and how quickly you need to undo a mistake.

Rolling

A rolling deploy replaces instances in batches. Take a share of the instances out of the pool, start new ones, wait until they are healthy, and repeat. Kubernetes Deployments do this with maxUnavailable and maxSurge. Capacity stays up if surge is set correctly.

The catch is that old and new versions run side by side during the rollout. The API must work with both. A rolling deploy is simple and needs no second fleet. But a bad build spreads across the fleet slowly, and it is hard to snap back unless you keep the previous set of pods ready.

Blue-green

Run two full environments. Blue is the current version and green is the new one. Deploy green, check it with synthetic or shadow traffic, then flip the load balancer to green. Rolling back is another flip.

The cutover and rollback are instant at the traffic layer, which is easy to reason about. The cost is roughly twice the compute during the transition. Long-lived connections and shared state need care. The flip does not migrate the database, so schema changes still need their own plan.

Canary

Send a small share of real production traffic to the new version, such as 1%, then 10%, 50% and 100%. Compare error rate, latency and business metrics against the baseline. Promote only when the gates pass. Otherwise shift traffic back.

A canary limits blast radius, and it uses real traffic patterns that staging never reproduces. It needs good metrics and enough traffic to show a signal. A canary that shares a cache or a worker pool with the baseline can poison the comparison.

Shadow

Duplicate production requests to the new version, and throw its responses away. This checks CPU use, parse errors and cache behaviour for risky changes without affecting users. It cannot check whether writes are correct. Shadowing a request that writes data writes it twice, so be careful with mutations.

Common mistakes

• Rolling deploys that break old and new pods together. Both versions run during the rollout, so the API must be compatible with both.
• Blue-green with one shared database and incompatible writers. The two stacks corrupt each other's data.
• Shadowing writes. The same change is applied twice.

Check yourself

1. In a rolling update, why must the API work with both versions? Answer: Old and new pods run at the same time for the length of the rollout.
2. What does blue-green cost during the transition? Answer: Roughly twice the compute, because both full environments are running.

Key takeaways

• Rolling is simple and cheap, but the versions overlap.
• Blue-green gives an instant cutover and rollback, at roughly double the compute.
• Canary limits blast radius and uses real traffic, but needs good metrics.
• Shadow tests the new version safely, but cannot validate writes.

Further learning

Next, see how a machine decides whether a canary should move forward: automated canary analysis.