Safe Terraform changes

One apply can delete a database. Safe workflows make that hard to do by accident. They run a plan for every change, check policy before anything applies, and add guards in the config that refuse to destroy stateful resources.

Walking through a pull request

1. On every pull request, run a format check, validate, and plan. Post the plan as a comment on the PR.
2. On merge to main, or after a manual approval, apply the saved plan.
3. Limit who can approve prod applies. Use short-lived cloud credentials through OIDC, not long-lived access keys stored in CI secrets.

Policy as code

Policy as code means a tool reads the plan and fails it if it breaks a rule. Open Policy Agent (OPA) with Conftest, HashiCorp Sentinel, and some vendor tools do this. A policy can block a plan that opens SSH to the internet, creates an unencrypted disk, or destroys a tagged prod database.

Start with a short denylist of high-severity rules. A long list nobody maintains catches nothing.

Guarding stateful resources

Put guards in the configuration itself:

resource "aws_db_instance" "primary" {
  # ...
  deletion_protection = true
  lifecycle {
    prevent_destroy = true
  }
}

prevent_destroy makes Terraform refuse any plan that would delete the resource. To remove the database, someone must first make a separate change that removes the guard. Pair it with cloud-side deletion protection and snapshots. For disks and buckets, turn on versioning or retention so a delete can be recovered.

Force-new changes

Some attribute changes update in place. Others force a replacement. Changing instance_class on RDS is often an in-place modify. Changing the identifier, or some engine options, can replace the database. A replacement creates a new instance, and it starts empty unless you restore data.

Read the provider docs for any attribute that holds data before you change it.

Common mistakes

• Applying from a laptop against prod. Use CI identities and approvals.
• Using long-lived keys in CI. Use OIDC so credentials expire.
• Writing a long policy list on day one. Start with the few rules that would hurt most.

Check yourself

1. What does prevent_destroy do? Answer: Terraform refuses any plan that would delete the resource, until someone removes the guard in a separate change.
2. Why is changing an RDS identifier risky? Answer: it is a force-new change, so Terraform replaces the database with a new one that starts empty unless you restore data.

Key takeaways

• Run plan on every pull request. Apply a saved plan after approval.
• Guard stateful resources in the config and in the cloud.
• Check policy before apply, and keep the denylist short.
• Watch force-new attributes on anything that holds data.

Further learning

Running Terraform in automation (HashiCorp)Common techniques for running Terraform in continuous delivery pipelines.Running Terraform in automation | Terraform | HashiCorp Developer

The lifecycle meta-argument (Terraform docs)How to configure the lifecycle meta-argument rules in Terraform--create_before_destroy, prevent_destroy, ignore_changes, replace_triggered_by, precondition, postcondition, destroylifecycle meta-argument reference | Terraform | HashiCorp Developer