Est.

Flaky Tests: What They Are and How to Handle Them in CI

A flaky test passes and fails on the same code. Until it is handled, a red build no longer means something broke, and a green one no longer means nothing did.

The Architecture Graph Editors · Updated

What a flaky test is

A flaky test is a test that can both pass and fail when nothing it depends on has changed: the same commit, the same test code, the same configuration. Run it ten times and you get a mix of results. The outcome depends on something the test does not control, such as timing, the order tests run in, shared state, the clock, the network or the machine it lands on.

That is different from a test that is simply broken (it fails every time) and from a test that fails because the code under it really changed. The defining property is non-determinism: the result is not a function of the code alone.

Why flaky tests matter more than they look

A CI run exists to produce one verdict per change: does this commit still do what the suite says it should? A flaky test attacks that verdict from both sides.

  • False red. A failure that has nothing to do with the change blocks a merge, sends someone (or a coding agent) to investigate code that is fine, and costs a full rerun.
  • Trained-in distrust. Once people learn that red often means "flaky", they rerun by reflex. The next real failure gets rerun too, and if it happens to pass, it merges.
  • Hidden regressions. A test that sometimes passes can mask a real intermittent bug in the product, such as a race condition, rather than in the test.
  • Noisy signals for everything built on top. Test selection, sharding and caching all assume a test's result reflects the code. Flakiness makes it impossible to tell whether a changed result came from the change or from chance.

This is why flaky tests should be handled on their own, in the open. Folding them into other mechanisms (skipping them during test selection, retrying them silently inside a shard) hides both the instability and whether the mechanism itself is correct. The site's piece on preserving the full-run verdict makes the same point for selective testing.

The usual causes

Research on open-source projects, starting with Luo et al.'s 2014 study "An Empirical Analysis of Flaky Tests", found that most flaky tests trace back to a small set of root causes, led by asynchronous waits, concurrency and test order dependency. In practice the list looks like this:

CauseWhat happensTypical sign
Asynchronous waitsThe test sleeps a fixed time or checks too early instead of waiting for the event it needs.Fails more on slower or busier runners.
ConcurrencyThreads, goroutines or async tasks interleave differently from run to run; a race or deadlock shows up only sometimes.Fails more under parallel execution or with more CPU cores.
Order dependenceA test relies on state another test left behind (a database row, a global, a mocked module), or breaks state another test needs.Passes alone, fails in the suite, or the reverse; changes when tests are shuffled or split across shards.
Time and datesAssertions on "now", time zones, daylight-saving changes, month ends or timeouts close to the real duration.Fails at particular times of day or dates, or on runners in another time zone.
External services and networkReal HTTP calls, DNS, package registries or third-party APIs that are slow, rate-limited or down.Failures cluster in time across many unrelated builds.
Resource leaks and limitsPorts, files, temp directories, memory or connections not released; a later test hits the limit.Fails late in long runs, or only on smaller machines.
Unseeded randomnessRandom inputs, unordered collections (hash map or set iteration) or generated IDs that occasionally hit an edge case.Rare failures with no pattern; can be reproduced with the same seed.
Environment differencesRunner images, tool versions, locale, CPU count or file-system behaviour differ between machines.Fails on one runner type or image and not another.

How to confirm a test is flaky

A single pass after a failure is a hint, not proof. Confirm it on purpose, on one fixed commit, so the code cannot be the variable:

  1. Pin the commit. Rerun the failing job on the exact same SHA. In GitHub Actions, "Re-run failed jobs" (or gh run rerun <run-id> --failed) reruns on the same commit.
  2. Repeat the test alone. Run just that test many times in a loop, for example go test -run TestName -count=100, pytest --count=100 with pytest-repeat, or Bazel's --runs_per_test=100. Failures here point at timing, concurrency or randomness inside the test.
  3. Run it inside the suite, in a different order. If it only fails alongside other tests, suspect order dependence. Shuffle the order (go test -shuffle=on, rspec --order random, pytest-randomly) and keep the seed of any failing run so it can be replayed. rspec --bisect can narrow down which other test causes the failure.
  4. Change the pressure. Run with more or less parallelism, on a smaller machine, or with the race detector (go test -race) to bring timing problems to the surface.
  5. Write down both outcomes. Record the commit, the runner, the seed and the failure message. A flaky test that cannot be reproduced on demand is still flaky; the record is what lets someone fix it later.

Retries: a measurement, not a fix

Most test frameworks can rerun a failed test automatically. Used well, a retry tells you a failure was not deterministic. Used badly, it turns every intermittent failure, including real product bugs, into a silent pass. The difference is whether a pass-on-retry is recorded and reported as flaky, or simply counted as green.

Where retries live in common tools. Option names change between versions; check each tool's current documentation.
ToolHow to retryReports pass-on-retry as flaky?
Playwrightretries in playwright.config or --retriesYes: the report marks such tests "flaky"
Jestjest.retryTimes(n) (jest-circus runner)Not by default; add reporting yourself
Vitestretry option or --retryCheck the reporter you use
pytestpytest-rerunfailures: --reruns nReruns are shown in the summary
Maven Surefire-Dsurefire.rerunFailingTestsCount=nYes: listed as flakes in the report
Gradleorg.gradle.test-retry plugin, maxRetriesCan fail the build on it with failOnPassedAfterRetry
Bazelflaky = True on the test target, or --flaky_test_attemptsYes: results show the test as flaky
RSpecrspec-retry gemDepends on configuration

Three rules keep retries honest:

  • Cap them low. One or two retries are enough to tell "sometimes" from "always". A test that needs five is not a passing test.
  • Count every pass-on-retry. Send it to a list, a dashboard or an issue tracker. If nobody can see how often each test needed a second try, retries are hiding failures.
  • Retry the test, not the job. Rerunning a whole job to get past one test reruns everything else too, and hides which test was the problem.

Quarantine without losing the signal

When a test is confirmed flaky and cannot be fixed today, quarantine it: take it out of the set of tests that decide the verdict, while keeping it running and visible. Martin Fowler's essay "Eradicating Non-Determinism in Tests" describes the practice. What makes quarantine safe is the policy around it:

  • It still runs. A quarantined test keeps executing on every build, so its results keep accumulating and the fix can be verified.
  • It has an owner and a ticket. Every quarantined test names who will fix it and links to the issue holding the evidence (commit, seed, failure message).
  • It has a limit. Cap how many tests may be quarantined at once, or for how long, so quarantine does not become a place tests go to be forgotten.
  • It is deliberate. A test enters quarantine through a reviewed change, never automatically and never because it slowed a merge down.

Be explicit about what quarantine costs: while a test is out of the verdict, the behaviour it checks is not protected. If it covers something important, fixing it should move up the queue.

Fixing flaky tests by cause

  • Waits: replace fixed sleeps with waiting on the condition itself (an element visible, a message received, a port open), with a timeout well above the normal duration.
  • Concurrency: make the ordering the test relies on explicit, run the race detector or thread sanitizer where the language has one, and fix the race in the product if that is where it is.
  • Order dependence: give every test its own setup and teardown, use fresh databases, schemas or transactions per test, and reset globals and module mocks between tests. Keep order randomization switched on so new dependencies show up early.
  • Time: inject a clock or freeze time in tests; set the time zone and locale explicitly in CI.
  • External services: replace real network calls with fakes, local containers or recorded responses inside unit and integration tests; keep the few true end-to-end tests separate and labelled.
  • Resources: allocate ports and temp directories dynamically and release them in teardown.
  • Randomness: seed every random source, print the seed on failure, and sort unordered collections before comparing them.
  • Environment: pin runner images and tool versions, and record them in the job so a difference can be seen in the logs.

After a fix, prove it the same way the flakiness was proved: run the test many times on the fixed commit, in the suite and alone, before it goes back into the verdict.

Flaky tests, sharding, test selection and coding agents

Anything that changes how a suite runs can expose flakiness that was already there. Splitting tests across shards changes which tests share a process and in what order, so order-dependent tests can start failing. More parallelism changes timing. Moving jobs to different runners changes CPU counts, images and caches. None of that creates a flaky test, but it is often when teams first notice one. The site's piece on cache warmth and test parallelism covers how these choices interact.

Test selection has a stricter problem: to argue that a subset of tests gives the same verdict as the full suite, the full suite's verdict has to be stable in the first place. A baseline with flaky tests in it cannot tell a selection mistake from noise. See what evidence selective testing needs and reachability-based test selection in GitHub Actions.

Coding agents raise the stakes. An agent told to "get CI green" can rerun a job until a flaky test passes, or edit the test until it stops failing, and both look like success. Give agents the same rules as people: a pass-on-retry is reported, not accepted; a test is never weakened or skipped to unblock a change; and a suspected flaky test is reported with its evidence for a person to decide on.

Frequently asked questions

What does "flaky test" mean?

A test that both passes and fails on the same code, with nothing changed between runs. Its result depends on something outside the test's control, such as timing, test order, shared state, the clock or the network.

Should I just retry flaky tests?

Retry to measure, not to hide. One or two automatic retries are a reasonable way to tell an intermittent failure from a deterministic one, as long as every pass-on-retry is recorded and reported as flaky and someone is responsible for fixing it.

Is it OK to delete or skip a flaky test?

Deleting it removes the protection it gave without fixing anything. Quarantine it instead: keep it running and visible, take it out of the verdict, give it an owner and a deadline, and bring it back once it passes reliably.

How do I find which tests are flaky?

Look for tests that failed and then passed on the same commit, either on a rerun or on a framework retry. Many frameworks and CI analytics products can report these; to confirm one, run it many times on a pinned commit, alone and inside the suite with a shuffled order.

Are there tools that detect flaky tests automatically?

Several test frameworks report pass-on-retry results (Playwright, Maven Surefire and Bazel among them), and a number of CI analytics vendors sell flaky-test detection. What each one detects and what it costs changes, so check the vendor's current documentation and pricing page.

Sources

Go deeper