Guide

Observability and reliability

Backups that restore

What makes a backup real, why restores must be timed, how point-in-time recovery works, and the GitLab 2017 outage that shows what untested backups cost.

140 of 162 · 3 min read

A backup you have never restored is a hope, not a plan. A real backup is automated, encrypted, stored away from the main account, and proven by a restore you have timed. This page covers those rules, plus point-in-time recovery (PITR), which lets you rewind a database to just before a bad write.

Concept

Point-in-time recovery combines a base backup with the database's write-ahead log (WAL). The WAL records every change in order. Replaying it up to a chosen timestamp rebuilds the data as it was at that moment.

The big idea

Think of a spare key. Owning one is not the same as knowing it turns the lock. You prove the spare works by trying it.

The spare-key analogy hides the cost of the test. Trying a key takes seconds. Restoring a large database can take hours, and a restore that works on 100 GB may not finish within the RTO for 10 TB.

Requirements

  • Automated. Human-scheduled backups get skipped.
  • Encrypted. At rest, with keys that are not only on the account that might be lost.
  • Off-account or off-region. Ransomware and a malicious destroy should not wipe the only copy.
  • Retained and immutable. Object-lock or WORM policies stop ransomware from deleting copies.
  • Tested. At least quarterly, restore into a clean environment, time it, and check the data.

Walking through a restore

After a restore test, write down these five things:

  1. Which snapshot or PITR timestamp you chose, and why.
  2. The wall-clock time until the first successful health check.
  3. How you verified the data: row counts, a checksum job or a set of known transactions.
  4. Which identities could delete this copy. Could one compromised role remove it?
  5. What broke in the runbook, and who owns the fix.

Record the measured RTO next to the objective. If the measured time is four hours and the objective is one, you do not have a one-hour RTO.

Warning

On January 31, 2017, GitLab lost data on its primary PostgreSQL database during an incident. The backup paths it relied on did not protect it. Some were incomplete, and some had not succeeded in practice. Some data was lost for good. Read the public postmortem for the checklist of "we thought we had copies."

Point-in-time recovery in practice

  • Know your provider's PITR window, which is often several days.
  • Turn on deletion protection, and separate the IAM permission to destroy from everyday access.
  • For logical corruption, restore to a side cluster before the bad event and copy the good rows back. Practice this.
  • Soft deletes and audit logs reduce how often you need a full restore.

Ransomware can reach snapshots if an attacker gets broad IAM. Immutable copies and break-glass credentials kept off the main identity provider protect against that. See IAM and secrets.

Common mistakes

  • Counting a replica as a backup. It replicates the delete, too.
  • Watching that backups exist, but not that they succeeded. Monitor success and age.
  • Keeping the only copy in the same account. One compromised role can delete everything.
  • Testing only small data. Test the size you would really restore.

Check yourself

  1. Why isn't a replica a backup? A replica copies every write, including a bad delete, so it cannot undo the damage.
  2. Why keep a copy in a second account? A ransomware attack or a destroy in one account cannot wipe the copy in the other.

Key takeaways

  • A backup counts only after a timed, verified restore.
  • Keep copies encrypted, immutable and off the main account.
  • PITR rewinds to a moment before a bad change.
  • Monitor backup success and age, not just that the job exists.

Further learning

Next, practice the whole recovery path in game days and region drills. The full GitLab postmortem is the reference for backup failure modes.

Postmortem of database outage of January 31 (GitLab)Postmortem on the database outage of January 31 2017 with the lessons we learned.GitLab

Disaster recovery and chaos engineering

RPO and RTO, backups that restore, game days and region drills, and deliberate failure injection, so confidence comes from measured recovery. Start here for the map of disaster recovery pages.

IAM and least privilege

How cloud identity works, why least privilege is a habit rather than a checkbox, and where the org-level guardrails sit. Start here for the map of IAM pages.

Secrets management and encryption

Keeping credentials out of code and data safe at rest and in transit with KMS, Vault and envelope encryption — plus how to catch leaks before they become incidents.