Rollback and roll-forward

When a release goes wrong, you have two options. Rollback restores the previous artifact or traffic shape. Roll-forward ships a fix on top of the broken version. Both are valid. The constraint that decides between them is how quickly you can restore service, and whether rollback is possible at all.

Choosing between them

Roll back when the previous version is intact and safe to run. Roll forward in three situations.

• The bad deploy already migrated data forward in a way the old code cannot read.
• Rolling back would bring back a security hole.
• The fix is a one-line config change and is already built.

During an incident, do not debate the philosophy. The incident response guidance is to mitigate first. Usually that means shifting traffic back, and then deciding.

Making rollback always possible

For binary changes, keep rollback available with four habits.

1. Keep the previous image digest immutable and in the registry.
2. Do not delete the old ReplicaSet or task definition until the new one has baked.
3. Avoid irreversible one-shot migrations in the same deploy as the app that needs them.
4. Prefer additive API changes. Remove a field only after its consumers are gone.

Walking through a bad release

A deploy adds a required column and makes the new code depend on it. The error rate rises five minutes later.

• If the old code still works with the new schema: shift traffic back to the old revision. Recovery takes seconds, and the schema change is harmless.
• If the migration dropped or rewrote data: the old code can no longer read the data. Roll forward with a fix, or restore from backup. Rollback alone will not help.

The difference is the schema, which is why the migration plan belongs in the release plan.

Common mistakes

• Debating the approach during the outage. Shift traffic first, and decide afterwards.
• Deleting the previous version too early. There is then nothing to roll back to.
• Shipping a destructive migration with the binary that needs it. Rollback is then no longer an option.

Check yourself

1. Why is rolling back a deploy that dropped a column often not enough? Answer: The previous binary may depend on that column, so the data must be restored or the code fixed forward.
2. When is roll-forward the right choice? Answer: When the bad deploy has already migrated data in a way the old code cannot read, or when rollback would bring back a security hole.

Key takeaways

• Rollback restores the previous version. Roll-forward fixes the broken one.
• Rollback is fast only when the old version can still run.
• Keep the previous digest and the old revision until the new one is proven.
• Make migrations expand-only, so rollback stays possible.

Further learning

Next, see how to change the schema without breaking the version that is still running: zero-downtime schema changes.