{"post":{"slug":"restore-drills-2026","title":"Restore Drills: Practising Recovery Before You Need It","excerpt":"Verified backups prove the data can come back. A drill proves your team can bring it back, under pressure, using a procedure that has been read by someone other than its author.","category":"code","date":"January 9, 2026","author":"ByteLedger Team","content":"<span class=\"dropcap\">A</span>utomated verification tells you the backup restores. It says nothing about whether your team can execute a recovery at three in the morning, with a partial outage, using a runbook written eighteen months ago by someone who has left.\n\nThose are different failures, and the second is the more common one during a real incident.\n\n## What changed in 2026\n\n- **Drills became a routine practice.** Regular recovery exercises moved from advanced operations into ordinary reliability work.\n- **The unfamiliar-operator principle spread.** Having someone other than the runbook author execute it became standard.\n- **Timing data became the main output.** Measuring each step rather than the total became recognised as more actionable.\n- **Automation reduced but did not eliminate the need.** Even where recovery is largely automated, the surrounding decisions remain human.\n\n## What a drill tests that verification does not\n\n| Question | Verification | Drill |\n| --- | --- | --- |\n| Does the backup restore? | Yes | Incidentally |\n| Is the runbook accurate? | No | Yes |\n| Can someone unfamiliar follow it? | No | Yes |\n| Do people have the access they need? | No | Yes |\n| How long does the whole process take? | Partly | Yes |\n| Are the decision points clear? | No | Yes |\n| Do communications work? | No | Yes |\n\nThe access question surfaces reliably. Credentials expired, permissions never granted to the people on call, a secret in a vault only one person can open, or a runbook referencing a system that was decommissioned.\n\nNone of that appears in an automated verification, and all of it stops a recovery cold.\n\n## Run it properly\n\n**Someone unfamiliar executes it.** The author knows what the runbook meant to say. Someone else discovers what it actually says. This single choice surfaces more gaps than any other aspect of the drill.\n\n**Follow the runbook literally.** If a step is ambiguous, stop and note it rather than filling in from knowledge. The ambiguity is the finding.\n\n**Time every step.** Not just the total. Where the time goes is consistently surprising — frequently more is spent finding the right backup, getting access, or deciding what to do than on the restore itself.\n\n**Do not fix problems silently.** Note them. The corrections to the runbook are the main output of the exercise.\n\n**Include the decisions, not just the mechanics.** Who authorises a restore? How is the target time chosen? Who tells customers? Those questions consume real time during an incident and are usually undocumented.\n\n## Scenarios worth drilling\n\nDifferent failures require different responses, and a single drill of one scenario does not prepare you for the others.\n\n**Full loss.** The primary is gone and must be rebuilt from backup. The classic scenario.\n\n**Accidental deletion.** Data must be recovered to a point in time while preserving subsequent legitimate writes. Considerably harder than a full restore, and much more common — see [point-in-time recovery](/blog/point-in-time-recovery-2026).\n\n**Partial corruption.** One table is damaged and the rest is fine. Tests whether selective restore is possible at all.\n\n**Failover.** The primary is unhealthy and a replica must be promoted. Different mechanism entirely — see [failover testing](/blog/failover-testing-2026).\n\nThe deletion scenario is worth prioritising because it is the most likely and the least practised. Most runbooks describe rebuilding from a full backup; few describe extracting a table's rows from a point-in-time restore without discarding a day of other work.\n\n## Common mistakes\n\n- **Discussing rather than doing.** A tabletop exercise is not a drill.\n- **The runbook author executing it.** Cannot find their own gaps.\n- **Fixing problems without recording them.** The findings are the point.\n- **Only drilling full recovery.** Deletion recovery is more likely.\n- **Not timing individual steps.** Hides where time actually goes.\n- **Skipping the access and decision parts.** Frequently the slowest.\n- **Doing it once and considering it done.** Systems and people change.\n\n## FAQ\n\n**How often should we drill?**\nQuarterly is a reasonable cadence, and after any significant infrastructure change. Less often and the runbook drifts; more often and it becomes routine rather than instructive.\n\n**Should we drill in production?**\nRestore *to* a separate environment rather than over production. The realism worth having is in the procedure and the people, not in risking live data.\n\n**What if the drill fails?**\nThat is a successful drill. Finding the problem during an exercise is the entire purpose, and a drill that reveals nothing usually means it was too easy.\n\n**Who should participate?**\nWhoever would actually be involved, including people outside the database team — whoever grants access, whoever communicates with customers, whoever makes the call to proceed.\n\n## Where to go next\n\nFor the automated half, read [backup verification](/blog/backup-verification-2026). For the recovery mechanism, [point-in-time recovery](/blog/point-in-time-recovery-2026), and for exercising promotion rather than restore, [failover testing](/blog/failover-testing-2026).","hue":308,"metaTitle":"Restore Drills: Practising Recovery Before You Need It","metaDescription":"Why restore drills test the runbook and the team rather than the backup, how to run one usefully, and what the timing data reveals.","keywords":["restore drill","disaster recovery exercise","runbook testing","recovery time objective","incident preparedness","game day","operational readiness","database operations 2026"],"tldr":["**Verification tests the backup; a drill tests the humans and the runbook** — different failures.","**Someone unfamiliar should run it** — the author of a runbook cannot find its gaps.","**Time every step** — where the time goes is usually surprising.","**Update the runbook during the drill** — the corrections are the main output.","**Skip:** an annual tabletop exercise instead of an actual restore. Discussing recovery is not practising it."],"readTime":"4 min read","wordCount":805},"related":[{"slug":"point-in-time-recovery-2026","title":"Point-in-Time Recovery: Undoing the DELETE You Regret","excerpt":"Replication copies your mistakes instantly. Point-in-time recovery is the only mechanism that lets you go back to just before someone ran a DELETE without a WHERE clause.","category":"code","date":"January 4, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":304},{"slug":"backup-verification-2026","title":"Backup Verification: A Backup You Have Not Restored Is a Guess","excerpt":"Backup jobs report success by writing a file. Whether that file can be restored into a working database is a different question, and it is the only one that matters.","category":"code","date":"January 6, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":313},{"slug":"failover-testing-2026","title":"Failover Testing: Finding Out Before Production Does","excerpt":"Automatic failover is configured and has never fired. Whether it works, how long it takes, and whether your application reconnects are all unknowns until you trigger it deliberately.","category":"code","date":"January 12, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":308}],"sameCategory":[{"slug":"bazel-build-explained-2026","title":"Bazel Explained: What Hermetic Builds Actually Cost","excerpt":"Bazel delivers reproducible, precisely cached builds and demands that you describe everything. Here is what you get, what it costs, and who should say no.","category":"code","date":"August 19, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":145},{"slug":"graphql-federation-2026","title":"GraphQL Federation: One Graph, Many Teams","excerpt":"Federation composes independently-owned subgraphs into a single API. Here is what it solves, the operational cost it adds, and when a simpler gateway is enough.","category":"code","date":"August 19, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":300},{"slug":"hermetic-builds-explained-2026","title":"Hermetic Builds Explained: Same Inputs, Same Output","excerpt":"A hermetic build depends only on declared inputs. Here is what that property gives you, the common leaks that break it, and how to move toward it incrementally.","category":"code","date":"August 19, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":130},{"slug":"monorepo-tooling-compared-2026","title":"Monorepo Tooling Compared: What You Actually Need","excerpt":"Monorepo tools solve task orchestration and caching, not repository layout. Here is what each tier provides and how to tell when you have outgrown the simple option.","category":"code","date":"August 19, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":150}],"newer":{"slug":"gift-splitting-explained-2026","title":"Gift Splitting: Doubling the Annual Exclusion as a Couple","excerpt":"A married couple can treat a gift from one spouse as coming half from each, doubling the annual exclusion. It requires a gift tax return, and consenting once applies to all gifts that year.","category":"finance","date":"January 9, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":285},"older":{"slug":"rope-scaling-explained-2026","title":"RoPE Scaling: Extending Context Beyond Training Length","excerpt":"A model trained on short sequences degrades sharply past that length. Rescaling its positional encoding lets it handle longer inputs without retraining from scratch.","category":"ai","date":"January 9, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":218}}