A flaky test is a test that passes sometimes and fails other times against the exact same code, with nothing meaningful changed between runs. It is not testing a real bug — it is testing something incidental: a race condition, a shared fixture, a network call that occasionally times out. The test itself is not lying about a code change; it is revealing that its own setup is unreliable.
What changed in 2026
- Flaky test detection became a built-in CI feature at major providers, automatically tracking pass/fail history per test and flagging inconsistent ones without any extra tooling setup.
- Automatic quarantine workflows matured — a detected flaky test gets pulled out of the required-to-pass set and filed as a ticket automatically, instead of silently blocking unrelated pull requests.
- Retry-until-green configurations came under scrutiny, after teams noticed blanket retries were hiding hundreds of genuinely flaky tests instead of surfacing them for a fix.
- Root-cause categories became standardized in tooling reports — timing, isolation, environment, and concurrency are now common labels, making it faster to triage a new flake.
The usual causes of flakiness
| Cause |
What is happening |
Typical fix |
| Timing / race conditions |
Test asserts before an async action finishes |
Await the actual condition, not a fixed delay |
| Shared state |
Tests run in parallel and collide on the same data |
Isolate fixtures per test, avoid shared globals |
| Real network calls |
An external service is slow, rate-limited, or down |
Replace with a mocked or local endpoint |
| Test order dependency |
A test relies on state left by a previous test |
Make every test independent and self-contained |
| Environment differences |
Passes locally, fails in CI, or the reverse |
Match environments; pin versions and locale/timezone |
Real hardware-level nondeterminism is rare in application test suites. Almost every flaky test traces back to one of these five categories.
Finding the actual cause
- Re-run the failing test in isolation, many times, before touching anything else. If it never fails alone, the cause is likely shared state or ordering.
- Check for unmocked network or filesystem calls. A test hitting a real, uncontrolled dependency is one of the most common sources — see what an API mock server is in 2026 for how to remove that dependency entirely.
- Look for fixed sleeps and timeouts (
sleep(500)) instead of waiting on an actual condition or event.
- Compare local and CI environments for differences in timezone, locale, parallelism, or available resources.
How to handle a flaky test without losing trust in CI
Quarantine it immediately: remove it from the required, blocking test set so it stops failing unrelated pull requests, but keep it running and visible so it is not forgotten. File a ticket the same day. A quarantined test with no ticket and no owner just becomes permanently ignored, which defeats the purpose of having it at all.
Common mistakes
Re-running a failed CI job until it goes green. This makes the symptom disappear without fixing the cause, and it can hide a very real, occasional production bug behind "it is just flaky."
Deleting the test instead of fixing the cause. Sometimes justified if the test has no real value, but often the underlying issue — a race condition, an unmocked dependency — will surface again somewhere else.
Ignoring flakiness because "it is probably nothing." Left unaddressed, a handful of flaky tests teaches an entire team to distrust red builds, which is far more expensive than the original bug the test was meant to catch.
FAQ
Is a flaky test the same as a broken test?
No. A broken test fails consistently and points at a real problem. A flaky test is inconsistent on identical code, which usually points at the test itself rather than the code under test.
Should flaky tests be deleted?
Only as a last resort, after investigation shows the test has little value or the flakiness cannot reasonably be fixed. Deleting without investigating just removes coverage without removing the underlying risk.
How do I stop a flaky test from blocking my team?
Quarantine it out of the required checks immediately, file a ticket, and keep it visible in reporting so it gets fixed rather than forgotten.
Can retries ever be an acceptable fix?
A single automatic retry for genuinely external, rate-limited services can be reasonable as a stopgap, but it should not be the default answer for internal test flakiness.
Where to go next