{"post":{"slug":"failover-testing-2026","title":"Failover Testing: Finding Out Before Production Does","excerpt":"Automatic failover is configured and has never fired. Whether it works, how long it takes, and whether your application reconnects are all unknowns until you trigger it deliberately.","category":"code","date":"January 12, 2026","author":"ByteLedger Team","content":"<span class=\"dropcap\">H</span>igh availability is configured. There is a standby, automatic promotion is enabled, and the documentation says failover takes thirty seconds. Nobody has ever tested it.\n\nWhen it fires for real, the database frequently does the right thing and the application does not — connections stay pinned to a host that no longer exists, the pooler holds stale connections, DNS caches an old address, and a thirty-second database failover becomes a fifteen-minute application outage.\n\n## What changed in 2026\n\n- **Deliberate failure injection became mainstream.** Testing failover on purpose moved into ordinary reliability practice.\n- **Client-side handling improved.** More drivers and poolers gained proper topology awareness and reconnection.\n- **The reconnection gap got documented.** Recognition that database failover and application recovery are different durations.\n- **Managed platforms exposed manual failover.** Triggering a controlled failover became a supported operation rather than a support ticket.\n\n## What actually breaks\n\n| Component | Common failure |\n| --- | --- |\n| Database promotion | Usually works |\n| DNS or endpoint update | Cached by clients |\n| Connection pool | Holds connections to the old host |\n| Application driver | Does not detect the change |\n| Retry logic | Absent or gives up too fast |\n| In-flight transactions | Fail; may or may not be retried |\n| Monitoring | Does not distinguish failover from outage |\n\nThe database layer is usually the reliable part — it is the component whose failover was designed and tested by its vendor. The gap is everything above it.\n\nConnection pools are the most common culprit. A pool holding connections to the old primary will keep handing them out until something forces it to notice. Whether your pooler detects the change, how quickly, and whether it drains cleanly are all things to establish by testing — see [connection pooling explained](/blog/connection-pooling-explained-2026).\n\n## Measure time to serving\n\nThe number that matters is not how long promotion takes. It is how long until your application is serving requests normally again.\n\nThose differ substantially. Promotion may take seconds while clients take minutes to notice, reconnect, and refill their pools.\n\nMeasure the whole sequence: failure detected, promotion started, promotion complete, first successful application query, error rate back to baseline. The gap between promotion complete and error rate normal is where the surprises live, and it is the part you can actually improve.\n\n## Test the failures that happen\n\nA clean shutdown is the easiest case and the least representative.\n\n**Hard failure** — the instance disappears without warning. Closer to a real hardware failure.\n\n**Hung primary** — the process is running and not responding. Harder than a dead one, because health checks may consider it alive and failover may not trigger at all.\n\n**Network partition** — the primary is healthy but unreachable from the standby. Tests split-brain protection specifically.\n\n**Slow degradation** — the primary is responding, slowly. The case where automatic failover most often fails to trigger and a human must decide.\n\nThe hung and degraded cases deserve most attention, because they are where automatic failover is least reliable and where the decision to intervene manually is hardest.\n\n## Check for data loss\n\nWith asynchronous replication, a promoted standby may be missing writes the failed primary had acknowledged — see [replication lag](/blog/replication-lag-explained-2026).\n\nThat window is real, and it should be a known quantity rather than a surprise. Testing gives you a measurement: write continuously, trigger failover, and compare what was acknowledged against what survived.\n\nIf the loss is unacceptable, that is an argument for synchronous replication and its latency cost — see [synchronous replication](/blog/synchronous-replication-2026). Making that a deliberate decision, informed by a measurement, is better than discovering the window during an incident.\n\n## Common mistakes\n\n- **Testing only clean shutdown.** The easiest and least realistic case.\n- **Measuring promotion time only.** Application recovery is the real number.\n- **Not testing the pooler's behaviour.** Where recovery usually stalls.\n- **Assuming drivers detect topology changes.** Many do not without configuration.\n- **No test of the hung-primary case.** Where automatic failover most often fails.\n- **Not quantifying data loss.** The window exists; measure it.\n- **Testing once.** Configuration and client libraries change.\n\n## FAQ\n\n**How often should we test?**\nQuarterly, and after any change to the database version, driver, pooler, or topology. Each of those can alter failover behaviour.\n\n**Should we test in production?**\nIdeally in a production-equivalent environment first, then in production during a low-traffic window once you are confident. Testing only in a simplified environment misses the client-side problems that matter most.\n\n**What if failover does not trigger?**\nThat is the finding. Health check configuration that does not detect the failure mode you injected is a real gap, and the hung-primary case is where it usually appears.\n\n**How do we test split brain?**\nPartition the network so both sides believe the other has failed, and verify that only one accepts writes. Consensus-based systems handle this by design — see [Raft consensus](/blog/raft-consensus-explained-2026).\n\n## Where to go next\n\nFor the replication choices that determine data loss, read [synchronous replication](/blog/synchronous-replication-2026) and [replication lag](/blog/replication-lag-explained-2026). For the recovery path when failover is not enough, [restore drills](/blog/restore-drills-2026).","hue":308,"metaTitle":"Failover Testing: Finding Out Before Production Does","metaDescription":"Why application reconnection breaks more often than database failover, how to measure real recovery time, and which failure modes to test.","keywords":["failover testing","automatic failover","high availability testing","connection recovery","split brain testing","chaos testing database","promotion time","database operations 2026"],"tldr":["**Application reconnection is what usually breaks** — the database fails over correctly and clients do not follow.","**Measure total time to serving, not time to promote** — promotion is frequently the fast part.","**Test the ugly failures, not just a clean shutdown** — a hung primary is harder than a dead one.","**Check for data loss after asynchronous failover** — the window is real and should be quantified.","**Skip:** trusting untested failover. Configured is not the same as working."],"readTime":"4 min read","wordCount":852},"related":[{"slug":"restore-drills-2026","title":"Restore Drills: Practising Recovery Before You Need It","excerpt":"Verified backups prove the data can come back. A drill proves your team can bring it back, under pressure, using a procedure that has been read by someone other than its author.","category":"code","date":"January 9, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":308},{"slug":"backup-verification-2026","title":"Backup Verification: A Backup You Have Not Restored Is a Guess","excerpt":"Backup jobs report success by writing a file. Whether that file can be restored into a working database is a different question, and it is the only one that matters.","category":"code","date":"January 6, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":313},{"slug":"point-in-time-recovery-2026","title":"Point-in-Time Recovery: Undoing the DELETE You Regret","excerpt":"Replication copies your mistakes instantly. Point-in-time recovery is the only mechanism that lets you go back to just before someone ran a DELETE without a WHERE clause.","category":"code","date":"January 4, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":304}],"sameCategory":[{"slug":"bazel-build-explained-2026","title":"Bazel Explained: What Hermetic Builds Actually Cost","excerpt":"Bazel delivers reproducible, precisely cached builds and demands that you describe everything. Here is what you get, what it costs, and who should say no.","category":"code","date":"August 19, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":145},{"slug":"graphql-federation-2026","title":"GraphQL Federation: One Graph, Many Teams","excerpt":"Federation composes independently-owned subgraphs into a single API. Here is what it solves, the operational cost it adds, and when a simpler gateway is enough.","category":"code","date":"August 19, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":300},{"slug":"hermetic-builds-explained-2026","title":"Hermetic Builds Explained: Same Inputs, Same Output","excerpt":"A hermetic build depends only on declared inputs. Here is what that property gives you, the common leaks that break it, and how to move toward it incrementally.","category":"code","date":"August 19, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":130},{"slug":"monorepo-tooling-compared-2026","title":"Monorepo Tooling Compared: What You Actually Need","excerpt":"Monorepo tools solve task orchestration and caching, not repository layout. Here is what each tier provides and how to tell when you have outgrown the simple option.","category":"code","date":"August 19, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":150}],"newer":{"slug":"exclusion-constraints-explained-2026","title":"Exclusion Constraints: Preventing Overlaps in the Database","excerpt":"Two bookings for the same room at overlapping times is a rule no unique constraint can express. An exclusion constraint enforces it, atomically, without application-level locking.","category":"code","date":"January 12, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":302},"older":{"slug":"kv-cache-quantization-2026","title":"KV Cache Quantisation: More Sequences in the Same Memory","excerpt":"At long context lengths the attention cache outgrows the model weights. Storing it at lower precision buys back concurrency, at a quality cost that is usually smaller than expected.","category":"ai","date":"January 12, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":205}}