High availability is configured. There is a standby, automatic promotion is enabled, and the documentation says failover takes thirty seconds. Nobody has ever tested it.
When it fires for real, the database frequently does the right thing and the application does not — connections stay pinned to a host that no longer exists, the pooler holds stale connections, DNS caches an old address, and a thirty-second database failover becomes a fifteen-minute application outage.
What changed in 2026
- Deliberate failure injection became mainstream. Testing failover on purpose moved into ordinary reliability practice.
- Client-side handling improved. More drivers and poolers gained proper topology awareness and reconnection.
- The reconnection gap got documented. Recognition that database failover and application recovery are different durations.
- Managed platforms exposed manual failover. Triggering a controlled failover became a supported operation rather than a support ticket.
What actually breaks
| Component |
Common failure |
| Database promotion |
Usually works |
| DNS or endpoint update |
Cached by clients |
| Connection pool |
Holds connections to the old host |
| Application driver |
Does not detect the change |
| Retry logic |
Absent or gives up too fast |
| In-flight transactions |
Fail; may or may not be retried |
| Monitoring |
Does not distinguish failover from outage |
The database layer is usually the reliable part — it is the component whose failover was designed and tested by its vendor. The gap is everything above it.
Connection pools are the most common culprit. A pool holding connections to the old primary will keep handing them out until something forces it to notice. Whether your pooler detects the change, how quickly, and whether it drains cleanly are all things to establish by testing — see connection pooling explained.
Measure time to serving
The number that matters is not how long promotion takes. It is how long until your application is serving requests normally again.
Those differ substantially. Promotion may take seconds while clients take minutes to notice, reconnect, and refill their pools.
Measure the whole sequence: failure detected, promotion started, promotion complete, first successful application query, error rate back to baseline. The gap between promotion complete and error rate normal is where the surprises live, and it is the part you can actually improve.
Test the failures that happen
A clean shutdown is the easiest case and the least representative.
Hard failure — the instance disappears without warning. Closer to a real hardware failure.
Hung primary — the process is running and not responding. Harder than a dead one, because health checks may consider it alive and failover may not trigger at all.
Network partition — the primary is healthy but unreachable from the standby. Tests split-brain protection specifically.
Slow degradation — the primary is responding, slowly. The case where automatic failover most often fails to trigger and a human must decide.
The hung and degraded cases deserve most attention, because they are where automatic failover is least reliable and where the decision to intervene manually is hardest.
Check for data loss
With asynchronous replication, a promoted standby may be missing writes the failed primary had acknowledged — see replication lag.
That window is real, and it should be a known quantity rather than a surprise. Testing gives you a measurement: write continuously, trigger failover, and compare what was acknowledged against what survived.
If the loss is unacceptable, that is an argument for synchronous replication and its latency cost — see synchronous replication. Making that a deliberate decision, informed by a measurement, is better than discovering the window during an incident.
Common mistakes
- Testing only clean shutdown. The easiest and least realistic case.
- Measuring promotion time only. Application recovery is the real number.
- Not testing the pooler's behaviour. Where recovery usually stalls.
- Assuming drivers detect topology changes. Many do not without configuration.
- No test of the hung-primary case. Where automatic failover most often fails.
- Not quantifying data loss. The window exists; measure it.
- Testing once. Configuration and client libraries change.
FAQ
How often should we test?
Quarterly, and after any change to the database version, driver, pooler, or topology. Each of those can alter failover behaviour.
Should we test in production?
Ideally in a production-equivalent environment first, then in production during a low-traffic window once you are confident. Testing only in a simplified environment misses the client-side problems that matter most.
What if failover does not trigger?
That is the finding. Health check configuration that does not detect the failure mode you injected is a real gap, and the hung-primary case is where it usually appears.
How do we test split brain?
Partition the network so both sides believe the other has failed, and verify that only one accepts writes. Consensus-based systems handle this by design — see Raft consensus.
Where to go next
For the replication choices that determine data loss, read synchronous replication and replication lag. For the recovery path when failover is not enough, restore drills.