Bootcamp hub · Prev: Module 6
Learning goals
- Explain backup vs snapshot vs replication in plain words, and why only one of them is actually a backup
- Run a restore drill on a lab machine and record how long it took, honestly
- Schedule a backup with a lock, a log, and a freshness check that proves it still ran last night
Required posts
Read these before the lab. They cover the commands this module assumes you already have working.
- Disk full triage — you will need to find space for a restore, and diagnose why a backup job silently stopped.
- Cron job not running — a lot of “my backups stopped” reports turn out to be a cron problem, not a backup problem.
- Wrap cron safely — the locking and timeout pattern this module’s lab uses.
- Bash logging & errors — a backup script that fails silently is worse than no backup script.
Mental model
Backups are worthless until restored. That sentence sounds obvious and gets ignored constantly, because writing a backup feels like progress and testing a restore feels like extra work. Do the extra work.
Three things get confused with “backup”, and only one of them is one:
- Snapshots, in the common local sense (LVM, ZFS, a filesystem snapshot), live on the same storage as the thing they protect. A snapshot on a failing disk fails with the disk, and ransomware that reaches the storage layer can reach it too. Cloud block-storage snapshots (EBS and similar) are a different case, since they live on separate storage and can serve as a real backup, but they still need the same retention and restore-testing discipline as anything else on this list.
- Replication copies changes to a second location in near real time. It protects against hardware failure. It does not protect against a bad decision: delete a table on the primary, and replication faithfully deletes it on the replica seconds later. Replication answers “what if this server dies”, not “what if I broke this data”.
- A backup is a copy that is stored separately, on a schedule, that you have actually restored at least once, with a retention window long enough that you notice a mistake before the last good copy expires.
Say the three qualities out loud until they are automatic: separate storage, tested restore, known retention. Miss any one of the three and you are relying on hope more than you think you are.
Lab
Do this on a lab VM, not production. The point is to feel the restore, not to protect anything real yet.
- Dump a lab database (
mysqldumporpg_dump) or tar a config tree, and write the output to a different disk or a different machine. If the copy lives next to the original, it is not a backup yet. - Delete a file, a table, or a row on purpose. Pick something you would actually miss, so the restore feels real.
- Restore from the copy and verify it: a checksum or row count you noted beforehand catches a corrupted or partial restore, but also spot-check the actual content, since a matching count alone does not prove the data is right.
- Write down how long the restore took, and what you had to look up to do it. That number is your real recovery time, not the one you assumed.
Most people discover their restore process during their first real incident. This lab is so that isn’t the first time.
What good looks like
- Job logs show a clear success line with a timestamp, not just “no errors” by omission.
- A status file or a metric proves the last successful run was recent — see the freshness-check pattern in wrapping cron safely.
- Someone other than you can follow your restore notes without asking you a question. If they can’t, the notes are a memory aid, not a runbook.
Interviewers ask about your last restore, not your backup schedule, because the schedule is easy and the restore is where people get caught out. Practice the answer with scenario drills.
FAQ
How long should this module take?
Plan one focused evening for reading plus one for the lab. Rushing past the restore drill defeats the point of the module — the value is in watching the restore actually work, or watching it fail and fixing why.
Do I need production access?
No. Use your lab. Practising backup and restore drills against production without a change window and permission is how backups themselves cause outages.