Automated verification tells you the backup restores. It says nothing about whether your team can execute a recovery at three in the morning, with a partial outage, using a runbook written eighteen months ago by someone who has left.
Those are different failures, and the second is the more common one during a real incident.
What changed in 2026
- Drills became a routine practice. Regular recovery exercises moved from advanced operations into ordinary reliability work.
- The unfamiliar-operator principle spread. Having someone other than the runbook author execute it became standard.
- Timing data became the main output. Measuring each step rather than the total became recognised as more actionable.
- Automation reduced but did not eliminate the need. Even where recovery is largely automated, the surrounding decisions remain human.
What a drill tests that verification does not
| Question |
Verification |
Drill |
| Does the backup restore? |
Yes |
Incidentally |
| Is the runbook accurate? |
No |
Yes |
| Can someone unfamiliar follow it? |
No |
Yes |
| Do people have the access they need? |
No |
Yes |
| How long does the whole process take? |
Partly |
Yes |
| Are the decision points clear? |
No |
Yes |
| Do communications work? |
No |
Yes |
The access question surfaces reliably. Credentials expired, permissions never granted to the people on call, a secret in a vault only one person can open, or a runbook referencing a system that was decommissioned.
None of that appears in an automated verification, and all of it stops a recovery cold.
Run it properly
Someone unfamiliar executes it. The author knows what the runbook meant to say. Someone else discovers what it actually says. This single choice surfaces more gaps than any other aspect of the drill.
Follow the runbook literally. If a step is ambiguous, stop and note it rather than filling in from knowledge. The ambiguity is the finding.
Time every step. Not just the total. Where the time goes is consistently surprising — frequently more is spent finding the right backup, getting access, or deciding what to do than on the restore itself.
Do not fix problems silently. Note them. The corrections to the runbook are the main output of the exercise.
Include the decisions, not just the mechanics. Who authorises a restore? How is the target time chosen? Who tells customers? Those questions consume real time during an incident and are usually undocumented.
Scenarios worth drilling
Different failures require different responses, and a single drill of one scenario does not prepare you for the others.
Full loss. The primary is gone and must be rebuilt from backup. The classic scenario.
Accidental deletion. Data must be recovered to a point in time while preserving subsequent legitimate writes. Considerably harder than a full restore, and much more common — see point-in-time recovery.
Partial corruption. One table is damaged and the rest is fine. Tests whether selective restore is possible at all.
Failover. The primary is unhealthy and a replica must be promoted. Different mechanism entirely — see failover testing.
The deletion scenario is worth prioritising because it is the most likely and the least practised. Most runbooks describe rebuilding from a full backup; few describe extracting a table's rows from a point-in-time restore without discarding a day of other work.
Common mistakes
- Discussing rather than doing. A tabletop exercise is not a drill.
- The runbook author executing it. Cannot find their own gaps.
- Fixing problems without recording them. The findings are the point.
- Only drilling full recovery. Deletion recovery is more likely.
- Not timing individual steps. Hides where time actually goes.
- Skipping the access and decision parts. Frequently the slowest.
- Doing it once and considering it done. Systems and people change.
FAQ
How often should we drill?
Quarterly is a reasonable cadence, and after any significant infrastructure change. Less often and the runbook drifts; more often and it becomes routine rather than instructive.
Should we drill in production?
Restore to a separate environment rather than over production. The realism worth having is in the procedure and the people, not in risking live data.
What if the drill fails?
That is a successful drill. Finding the problem during an exercise is the entire purpose, and a drill that reveals nothing usually means it was too easy.
Who should participate?
Whoever would actually be involved, including people outside the database team — whoever grants access, whoever communicates with customers, whoever makes the call to proceed.
Where to go next
For the automated half, read backup verification. For the recovery mechanism, point-in-time recovery, and for exercising promotion rather than restore, failover testing.