Flakiness, Coverage, and Suites That Survive

A flaky test is one that passes and fails on the same code. It is a worse problem than a missing test, and the reason is about people rather than software.

A suite's value rests entirely on one inference: red means broken. Flakiness severs it. Once a team learns that red sometimes means nothing, the rational response to a red build is to re-run it, and that response is indistinguishable from the response to a genuine regression. The suite has not become 90 percent as useful; it has stopped being evidence.

Key idea: flakiness does not degrade a suite proportionally. It attacks the inference the suite exists to support, so a small number of unreliable tests can neutralise thousands of good ones by teaching everyone to ignore failures.

Worse, the damage is self-reinforcing. Re-running until green is cheap per incident and quietly becomes policy, and once "click retry" is muscle memory, a real regression rides through on the second attempt with nobody having made a decision.

Individual flakiness rates that sound negligible become fatal once you multiply them across a suite. If each test independently passes with probability p, a suite of n tests goes green with probability p to the power n.

The independence assumption is a simplification, real flaky failures cluster by cause, but it errs in the reassuring direction for the argument being made: correlated failures make individual bad builds worse, not rarer.

Flaky tests are not mysterious. They come from a short list of sources, and each has a structural fix rather than a retry.

Two of these deserve extra attention because they are so often misdiagnosed. Test-order dependence is usually blamed on "the CI machine", when the real cause is leaked state between tests: randomising test order locally exposes it in one run. And sleep(2) in a test is not a fix for a race; it is a bet that two seconds is always enough, which fails whenever the machine is busy, so the test gets flakier exactly when the pipeline is under load.

Ad hoc responses to flakiness always converge on blanket retries, which preserve the green build and destroy the information. A policy with an explicit quarantine keeps both.

When a test fails, the first question is whether it fails deterministically on the same commit. If it does, it is a real failure and the build stops, which is the entire point of the suite.

If it does not reproduce, the test is flaky, and it moves to quarantine: still executed, still reported, but no longer able to block the build. Crucially, quarantine is a queue, not a graveyard, so each quarantined test carries an owner and a deadline. Fix it and it returns to the blocking suite. Let the deadline pass and it is deleted, because a test nobody will fix is a maintenance cost with no benefit, and pretending otherwise is how suites rot.

The rule that makes the whole thing work: the blocking suite must be trusted absolutely. Any test allowed to block the build must be one whose red is always worth stopping for, which means the quarantine list is allowed to be embarrassing, and the main suite is not.

The standard metric for suite quality is line coverage, and the research on it is more specific than either its fans or its critics usually report.

Inozemtseva and Holmes studied this directly in Coverage Is Not Strongly Correlated With Test Suite Effectiveness at ICSE 2014, generating 31,000 test suites from five large open-source Java projects of around 100,000 lines each, and measuring each suite's coverage against its ability to detect faults. Their finding: coverage and effectiveness correlate moderately to strongly when you ignore suite size, but the correlation drops to low or moderate once the number of tests is controlled for, and stronger coverage criteria than line coverage do not give better insight. The paper won an ACM Distinguished Paper award and was recognised as an ICSE most influential paper a decade later.

Read carefully, that is not "coverage is useless". It is that much of coverage's apparent predictive power is a proxy for simply having more tests.

Gotcha: coverage measures execution, not verification. A test that calls a function and asserts nothing raises coverage exactly as much as one that checks every output. This is why coverage targets are so easy to satisfy without improving anything, and why a mandated percentage reliably produces assertion-free tests written to clear it.

Coverage remains useful when read as a map rather than a score.

The honest uses: find the untested regions and judge them individually, since "the payment module is at 12 percent" is actionable information; and watch the diff, because coverage on newly changed lines answers a question you actually care about, unlike a whole-repo percentage dominated by code nobody has touched in years.

The dishonest use is the target. A number in a gate converts into the cheapest tests that satisfy it, which are assertion-light tests over easy code.

The stronger signal is mutation testing, which measures verification rather than execution. The tool makes small changes to your code, flipping a comparison, changing a constant, removing a call, and re-runs the suite. If the tests still pass, that mutant "survived", meaning nothing in your suite noticed the behaviour change:

# original
if age >= 18:
# mutants a tool would generate
if age > 18:    # boundary
if age >= 19:   # constant
if True:        # condition removed

Surviving mutants point at exactly the assertions you are missing, and a test that raises coverage without checking anything kills no mutants at all. The cost is real, since the suite runs once per mutant, so it is usually applied to critical modules rather than a whole repository, which is the right place to spend it anyway.

Suite runtime is a correctness property, not a convenience. Past roughly ten minutes, developers stop waiting for results and start context-switching, and the feedback loop the suite exists to provide is gone even though every test still passes.

The levers, in the order they usually pay off:

• Parallelism. Nearly free if tests are independent, and it exposes hidden shared state immediately, which is a diagnosis, not a setback.
• Tiering. A fast suite on every commit, the slow tiers, end-to-end and integration against real services, on merge or on a schedule. Most teams get most of the value from this alone.
• Test selection. Run only tests affected by the changed files. Powerful at scale and dangerous if the dependency graph is wrong, so treat a full run on merge as the safety net.
• Fixing the slowest tests. Runtime is usually dominated by a handful of tests doing something silly: a real sleep, an unindexed query, a container started per test rather than per suite. Profile before optimising.

In practice: track suite runtime and flake rate as first-class metrics with owners, alongside pass rate. A suite whose runtime doubles every quarter has a deadline, and the team that measures it can act a year before the team that does not.

Pulling the four lessons together, the through-line is that a test suite is a product with users, a cost of ownership, and a failure mode of its own.

Its users are developers making changes, and what they need is a fast, trustworthy answer to one question: did I break something? Everything that damages that answer, flakiness, slowness, tests coupled to implementation, doubles encoding wrong beliefs, is a defect in the product, whatever the pass rate says.

The practices that follow are unglamorous and compound:

• Spend the testing budget on risk, not on uniformity of coverage.
• Assert on behaviour, so refactoring stays cheap and red keeps meaning something.
• Double at boundaries you own, and keep at least one path that touches reality.
• State laws where laws exist, and let a generator search the space you cannot imagine.
• Treat flakiness as a build-stopping defect, because it attacks the inference everything else rests on.
• Read coverage as a map, and use mutation testing where the stakes justify it.

Key idea: the aim was never a number. It is that a developer who did not write this code can change it with confidence, and find out in minutes whether they were wrong. Every technique in this course is in service of that sentence, and any practice that stops serving it, however orthodox, has become overhead.