The fix for a flaky test is never "run it again." A test that passes on one run and fails on the next, against identical code, is telling you that something in its setup is nondeterministic: a race against an async call, state leaked from another test, or a dependency that occasionally stalls. Chasing that down takes a repeatable process, not a hunch. This playbook is the diagnostic sequence for finding the exact cause and closing it with a real code fix, not a retry wrapper that just delays the next failure.
What changed in 2026
- CI providers now track pass/fail history per test automatically, surfacing a flakiness score without any extra tooling setup on top of the pipeline you already have.
- Test runners ship native flaky-detection modes. Retry-with-reporting in Vitest,
cargo-nextest's retry-and-flag behavior, and pytest-rerunfailures in analysis mode all now distinguish "passed on retry" from "passed clean," instead of quietly treating both the same.
- Blanket retry-until-green configurations came under real scrutiny after teams found they were masking hundreds of genuinely broken tests instead of surfacing them.
- Deterministic-time and seeded-random helpers became default scaffolding in most test templates, closing off two of the most common flakiness sources before a test is even written.
The diagnostic process
- Reproduce in isolation. Loop the single failing test 50 to 100 times on its own (
--repeat, -count=100, or your runner's equivalent). If it never fails alone, the cause is almost certainly shared state or ordering, not the test's own logic.
- Reproduce under real parallel load. Run the full suite repeatedly with your normal worker count. Failures that only appear here point at fixture collisions or a database or cache shared across workers.
- Bisect the suspects. Disable one variable at a time: drop to a single worker, freeze the clock, stub the network. Whichever change makes the failure disappear tells you the category.
- Match the signature to a cause using the table below.
- Apply the category-specific fix, not a generic timeout increase.
- Gate the fix with repeat runs — rerun the new version 50-plus times in CI before marking it resolved and removing it from quarantine.
Matching symptoms to fixes
| Symptom |
Likely cause |
Fix |
| Fails only in CI, never locally |
Resource contention across parallel workers |
Isolate fixtures per worker; reduce concurrency to bisect |
| Fails more often under full-suite load |
Shared mutable state (DB row, singleton, module cache) |
Per-test transaction rollback or uniquely generated fixture data |
| Intermittent timeout or hang |
A real, unmocked network or filesystem call |
Stub the dependency; replace a fixed sleep with an awaited condition |
| Passes alone, fails inside the suite |
Test order dependency |
Randomize execution order in CI to force the coupling to surface |
| Fails near midnight, month-end, or across timezones |
An unfrozen clock or DST/locale assumption |
Freeze or inject the clock; pin timezone and locale in CI |
Two fixes cover a large share of real-world flakiness:
// BAD: a fixed delay hopes the async work finished in time
await sleep(500);
expect(ui.textContent).toBe("Loaded");
// GOOD: wait on the actual condition, however long it takes
await waitFor(() => expect(ui.textContent).toBe("Loaded"));
// BAD: module-level state leaks between tests that run in the same process
let cache = {};
test("populates on first call", () => { cache.x = 1; /* ... */ });
test("starts empty", () => { expect(cache.x).toBeUndefined(); /* fails if run second */ });
// GOOD: reset shared state before every test
beforeEach(() => { cache = {}; });
Common mistakes
Retrying until CI goes green. This deletes exactly the signal you need — the failure pattern — without touching the underlying race condition, and it can hide a real intermittent production bug behind "it is just flaky."
Increasing a timeout instead of awaiting a condition. A longer timeout only raises the load or latency threshold at which the same race reappears; it does not remove the race.
Sharing one database or fixture set across parallel test workers. Two workers touching the same row is one of the most common causes of "fails only in CI," and it gets worse, not better, as you add more parallelism.
Quarantining without a ticket and an owner. A test pulled from the required check with no tracking simply becomes permanently ignored, which defeats the point of investigating it at all.
FAQ
Should I just increase the test timeout?
Only as a temporary stopgap while you investigate, never as the final fix. If the underlying cause is a race condition, a longer timeout just narrows the window in which it shows up.
How many repeat runs prove a fix actually worked?
Fifty to a hundred consecutive clean runs is a reasonable bar for a test that failed roughly one time in twenty. Scale the number up for rarer failures — a test that flaked once in two hundred runs needs proportionally more repeats to trust a fix.
Is randomizing test execution order overkill?
No — it is one of the cheapest ways to force hidden order dependencies to surface in CI instead of in front of a customer. Most major test runners support it as a flag.
Can a flaky test be pointing at a real production bug?
Yes, and this is the case teams miss most often. A race condition in a test is frequently the same race condition that occasionally corrupts data or drops a request in production, just easier to trigger under test load.
When is deleting a flaky test the right call?
Only after investigation shows the test covers little real risk and the flakiness cannot reasonably be fixed. Deleting without investigating removes coverage without removing the underlying risk it was pointing at.
Where to go next
A flaky suite and a noisy incident channel share the same root skill: isolating a variable under pressure instead of guessing. See how to debug production incidents in 2026 for that process applied to live systems, static analysis tools compared for 2026 for catching the bugs that cause flakiness before they ship, and blue-green vs canary deployment in 2026 for keeping a shaky pipeline from blocking a safe release.