Craft of code
Flaky test policy
A flake fails without a code change and trains the team to ignore red CI. Quarantine it the same day; fix or delete it within days.
A flaky test is one that fails without a code change. Flakes train the team to ignore red CI — the most expensive cultural failure a suite can cause. Treat them as P0.
Policy that works: quarantine a flake immediately, fix or delete within days, never "retry until green" as the permanent solution. A quarantined job is a temporary hospital with owners and tickets, not a permanent junk drawer.
Concept
Shared mutable fixtures and sleeps are common flake factories. Prefer the isolation habits in testing behavior and the cost model in test levels.
Causes and fixes
| Cause | Fix |
|---|---|
| Time / clocks | Inject Clock; freeze time in tests |
| Ordering | No shared mutable globals; isolate DB per test |
| Concurrency | Deterministic locks or single-thread test runs for that case |
| Network | Fake or record/replay; never hit live APIs in unit CI |
| Sleeps / waits | Wait on conditions, not sleep(500) |
| Random ports / files | Temp dirs; allocate ports dynamically |
When you inherit a bad suite, do not rewrite it wholesale. Apply the same strangler mindset as legacy refactors: add good tests on the path of current features, quarantine flakes, delete worthless tests when you touch their code.
A healthy CI pipeline usually stages a fast gate (unit + lint + types in under a few minutes on every push), then integration with shards (tens of minutes max for the default PR check), then nightly or pre-release E2E and mutation on hot packages. Fail the build on new flakes in the default gate.
Document how to run the smallest relevant subset locally (pnpm test path/to/file) so people actually run tests before push. A suite that only runs in CI will be developed blind and stay brittle.
When talking to PMs and managers, translate: "This flake burns ~Z engineer-hours/week in retriggers; fixing it is capacity, not polish." Same facts, outcome language.
Common mistakes
- Retry until green as the permanent fix. You trained the team to ignore red.
- Leaving flakes red for weeks. Nobody trusts CI.
- A quarantine lane with no owner. It becomes a junk drawer.
sleep(500)to "stabilize" a test. Wait on the condition.
Check yourself
-
A test failed on main with no diff. What do you do today? Quarantine it with an owner. Fix or delete within days. Do not merge retries as the policy.
-
Why is
sleep(500)a flake factory? You guessed a duration. Slow CI loses the race. Wait until the condition is true.
Key takeaways
- Flakes are cultural debt. Quarantine and fix; do not normalize retries.
- Isolate time, data, network, and files.
- Fail the default gate on new flakes.
- Keep local commands short so people run tests before they push.
Further learning
Software Engineering at GoogleAn open-source collection of core C++ library codeabseil.ioNext, extend confidence past CI with testing in production.
Related
Testing strategy
What to test, at which level, and how to keep a suite fast, reliable, and worth the maintenance — from pyramid trade-offs to production safety nets.
Test levels
Unit, integration, contract, and end-to-end tests buy different confidence per minute. Pick with cost, not habit.
Continuous integration and the software supply chain
Fast, trustworthy pipelines - build, test, artifact, scan and sign - plus DORA metrics and supply-chain controls that keep trunk green.