A backup you haven't restored is a guess

Every team has backups. Fewer teams have restores. The difference shows up at the worst possible moment: a dropped table, a corrupted volume, a server that won’t boot. That’s when you find out whether the nightly job has been producing anything you can actually use.

Here are the ways it usually goes wrong, and the habits that prevent each one.

The backup copies files that were changing

Copying a database’s data directory while the database is running gives you a snapshot of files mid-write. Sometimes it restores fine. Sometimes it doesn’t, and you won’t know which until you try.

Habit: back up databases with their own tools. pg_dump for Postgres, mysqldump --single-transaction for MySQL and MariaDB, a fresh RDB snapshot for Redis. These produce consistent backups while the database keeps serving.

The backup lives next to the thing it protects

A backup on the same server, or in the same account at the same provider, protects you from some failures and not others. A compromised account or a provider-wide incident takes both.

Habit: follow the old 3-2-1 rule in spirit: keep more than one copy, on more than one kind of storage, with one copy somewhere else entirely. Object storage at a different provider is cheap, and S3-compatible APIs make it easy.

Nobody can decrypt it

Encrypting backups is the right call. Storing the only copy of the key on the server you’re backing up is not.

Habit: keep a copy of every repository key outside the system it protects, in your password manager or with your other recovery documents. Test that you can find it without logging into production.

The restore has never been run

This is the big one. A backup job that exits 0 tells you it wrote something. It doesn’t tell you that what it wrote is complete, readable and recent.

Habit: restore regularly, automatically. Pick the latest snapshot, restore it into a throwaway container, and check that it can be read. For a database, start it and run a query against a table you know should have rows. Do it weekly. Alert if it fails.

$ restic restore latest --target /tmp/restore-check
$ docker run --rm -v /tmp/restore-check:/restore postgres:16 pg_restore --list /restore/dump.pgc

Retention that matches reality

Keeping every nightly snapshot forever gets expensive. Keeping only the last seven means a problem noticed on day eight can’t be fixed.

Habit: keep a few dailies, a few weeklies and a year of monthlies, and prune the rest automatically. Something like 7 daily, 4 weekly and 12 monthly snapshots covers most “when did this break?” questions with 23 snapshots. With deduplication, that’s often only a few times the size of the data itself.

Make it boring

The goal is for restores to be boring: a routine you’ve done many times, not an improvised procedure at 3 a.m. Consistent dumps, an off-site copy, a key you can find, and a test restore every week will get you most of the way there. Whatever tool you use, make sure it does those four things.