Dynamic config and blast radius

Feature-flag systems and remote config, such as a config service or a hot-reloaded file in object storage, share one property. A small write can change every process within minutes. That is the point of a kill switch. It is also why config pushes appear in postmortems as often as bad deploys.

Static versus dynamic

A static environment variable changes only on redeploy. That is slower, but safer and easier to reason about during an incident. A dynamic flag changes instantly. That is faster mitigation, and faster footguns.

Put rarely changed, high-blast-radius settings behind stricter controls than an experiment flag. Examples are the maximum connection pool, cluster membership and whether a region accepts traffic.

Walking through a safe push

Each step guards against one kind of failure. Validation rejects a malformed value before it leaves the console. The canary limits the damage to one region or 1% of instances. Watching SLOs decides whether to ramp or revert. The audit log records who changed what.

Patterns that keep blast radius small

1. Progressive delivery for config. Ship the change to 1% of instances, or to one region, first. This is the same discipline as a canary deploy.
2. Schema and validation. Reject unknown keys, out-of-range numbers and empty critical strings at the config service, not only in the app.
3. Versioning and rollback. Every snapshot has an ID. Reverting means restoring the previous ID, not editing by hand under pressure.
4. Authorisation and audit. Limit who can change production flags, require two people for high-impact keys, and log every change.
5. Safe defaults in code. If the flag service is down, the app uses its last-known or compile-time defaults. Usually that means "feature off".

Common mistakes

• Pushing a high-impact value to every region at once. Failures then arrive everywhere at once.
• Validating only in the app. A bad value is then accepted by the console and reaches every process.
• Defaulting to on when the flag service is down. An outage in the control plane then turns features on.

Check yourself

1. Why canary a config change, when it is "just a setting"? Answer: A setting can change the whole system's behaviour, so a 1% rollout shows the damage before it spreads.
2. The flag service is unreachable. What should the app do? Answer: Use its last-known or compile-time defaults, which should be safe, usually feature off.

Key takeaways

• Dynamic config is a production control plane, and it fails like one.
• Validate values at the source, and reject ones that are unknown or out of range.
• Canary config changes, version them and make rollback a single step.
• Audit every change, and default to safe values when the control plane is down.

Further learning

When a config change does cause an incident, the next step is the response process: incident response. For how the same staged approach applies to code, see deployment strategies.