Rotating secrets without an outage

Rotation replaces a secret with a new value and retires the old one. It fails when every consumer has to switch at the same instant. The fix is to let the old and new values work together for a short window. Then consumers move over one by one.

The big idea

Think of changing the locks on a house. You cut the new key, and for a day both keys open the door. Each family member switches to the new key. Once everyone has switched, you remove the old lock.

Where the analogy breaks: you must know who holds a key. For secrets, that means an inventory with an owner and a maximum age for each one. An unknown consumer is the one that breaks at the cutover.

Walking through a Postgres rotation

This rotation takes about 15 minutes, using a dual-credential setup.

t=0     Set password B for the database user. Accept A and B for now,
        using a dual-user setup or app-side logic.
t=1-10  Roll the pods. Each pod fetches the latest secret on boot.
t=10    Check the auth log. Confirm nothing logs in with password A.
t=15    Revoke A. Mark the rotation as done.

Step t=10 is the signal that makes step t=15 safe. You do not revoke on a timer alone.

Patterns that avoid pages

• Dual credentials. The database accepts two passwords. Apps move to the new one, then the old one is revoked.
• Token minting. Apps never see a shared password. Vault or IAM issues each instance its own credential with a time limit.
• Blue-green secrets. New deployments get the new secret through External Secrets. Old replicas drain.
• Short-lived certificates. Certificates last hours or days, from Vault PKI or SPIFFE. Revocation then mostly means waiting for expiry.

Common mistakes

• Caching a password forever in memory. After rotation, every cached copy fails at once. Refresh on an auth failure, or before the time limit.
• Rotating without knowing consumers. Without an owner tag, you find the consumers by breaking them.
• Revoking the old value too early. Check the auth log first.

Check yourself

1. Why does dual-credential rotation avoid downtime? Answer: both old and new values work during the window, so consumers can switch one at a time.
2. What tells you it is safe to revoke the old password? Answer: the auth log shows no logins with the old value for the whole window.

Key takeaways

• Let the old and new values work together during a window.
• Apps must refresh secrets. Caching them forever turns rotation into an outage.
• Short-lived certificates make rotation mostly a matter of waiting.
• Next, leak response covers what to do when a secret escapes anyway.

Further learning

Dynamic secrets for database credentials (HashiCorp Vault)Dynamically generate, manage, and revoke database credentials that meet your organization's password policy requirements.Dynamic secrets for database credential management | Vault | HashiCorp Developer