Flaky tests

A flaky test sometimes passes and sometimes fails, with no change to the code. Flakes destroy trust. Engineers re-run until the build goes green, learn to ignore red builds, or disable whole suites. Treat a flaky test as an incident in the pipeline, not as background noise.

Walking through the response

Handle each flaky test with four steps, in order.

1. Detect. Track pass and fail results by test name across commits. A test that fails repeatedly on commits that were otherwise green is flaky.
2. Quarantine. Move it out of the blocking path into a job that still runs and logs results. Do not delete it silently. Quarantine without an owner is how suites rot.
3. Own. Every quarantined test has a team and a deadline. Put the count of quarantined tests on a dashboard, next to build time.
4. Fix the cause. Most flakes come from a few sources.
   • Sleep-based waits instead of explicit synchronisation.
   • Shared mutable state between tests.
   • Assertions that depend on order.
   • Reliance on the wall clock.
   • Network calls to the public internet.

Prefer deterministic fakes and explicit synchronisation over waits and retries.

Common mistakes

• Retrying until green. The pass is real, but the test is still broken.
• Quarantining with no owner. The quarantine becomes permanent.
• Deleting a flaky test. You lose the coverage and hide the problem.

Check yourself

1. Why is silent auto-retry harmful? Answer: It hides the flake, so the team stops trusting red builds and the flake keeps spreading.
2. What is the first step for a new flaky test? Answer: Detect it by tracking results per test name, then move it out of the blocking path.

Key takeaways

• A flaky test erodes trust in the whole pipeline. Treat it as an incident.
• Detect, quarantine, assign an owner, then fix the root cause.
• Replace sleeps and shared state with explicit synchronisation and fakes.
• Retry at most once, and measure every retry.

Further learning

Next, make sure the artifacts the pipeline produces can be trusted: supply-chain security.