Stop Guessing Whether Your Test Is Flaky. Run It Five Times.
A simple discipline for separating real flakes from real bugs — and the four categories most of them turn out to fall into.
Every QA team has this conversation:
“That test? Yeah, it’s flaky.”
Thanks for reading! Subscribe for free to receive new posts and support my work.
On what evidence? Usually: vibes. Sometimes: one or two failures someone remembers. Almost never: actual repro data. The label sticks. The test goes onto the quarantine list. The team moves on.
Three months later, the test is still in quarantine. Nobody’s looked at it. Sometimes it was actually catching a real intermittent bug. Sometimes it really was flaky but for a reason a five-minute investigation would have fixed. Either way: the quarantine list grows, the coverage shrinks, and confidence in CI erodes by another tick.
I built a tiny harness to stop having this conversation.
It does one thing: take a test name, run it five times in isolation, count the results, and tell you what it actually is. About fifty lines of script, one clear verdict. No more vibes.
This article is that harness, the four categories most flaky tests turn out to fall into, and what I learned about the difference between “flaky” and “we haven’t looked at it.”
What “Flaky” Actually Means
Let’s get the definition out of the way, because the imprecision is half the problem.
A test is flaky if, given the same inputs and environment, it produces different results across runs.
That’s it. Not “fails sometimes.” Not “is annoying.” Different results across runs.
The distinction matters because there are three states a test can be in, and they look identical from a single run:
- STABLE — runs always pass.
- BROKEN — runs always fail.
- FLAKE — runs sometimes pass and sometimes fail.
The only way to distinguish them is to run the test more than once. Yet most flake decisions get made from a single failure. That’s the bug in our process I want to fix.
The Five-Run Discipline
The harness has one job: run a target test five times in isolation, count the outcomes, and emit a verdict.
5/5 pass → STABLE (false alarm — whatever you saw was likely an env hiccup)
0/5 pass → BROKEN (it's not flaky, it's failing - go find the bug)
1–4 pass → FLAKE (now we have evidence)Why five? Because three is too few to be confident, and ten is too slow to fit into the “I noticed something weird” window before you context-switch. Five is the round number that catches the long tail without making you wait twenty minutes. Pick a different number if you want — the point is some number, applied consistently, before anyone gets to use the word flaky out loud.
The “in isolation” part matters more than people think. If your test runs alongside other tests, you’ve conflated two questions: does this test work and does this test interact with its neighbors. Both questions are interesting. They’re not the same question. Run alone first.
Once You Know It’s a Flake — What Kind?
When the verdict comes back FLAKE, you’ve gained evidence. You haven’t gained understanding. The next question is why — and in my experience, around 95% of real flakes fall into four buckets.
TIMING. A wait isn’t long enough, or there’s a race between two async events. Symptom: “passes on my laptop, fails on the slow CI box.” Fix: explicit waits on observable conditions. If your test contains Thread.sleep(...), you have this problem until proven otherwise.
ORDERING. The test depends on previous tests leaving the world in a particular state. Symptom: passes when run alone, fails when run inside the suite. Fix: own your preconditions; never rely on a sibling test’s leftovers.
ENV_LEAK. A prior run of the same test left state behind, and the second invocation trips over it. Symptom: first run passes, subsequent runs fail. Fix: proper teardown. If your test creates an entity, your test deletes that entity — no matter how the test exits.
DATA_RACE. Two parallel test threads share a resource: a database row, a user account, a configuration toggle. Symptom: fails when concurrency > 1, passes when concurrency = Fix: scope data per-test, or single-thread the offender.
The harness can poke at these. After the initial 5-run isolated pass:
- Re-run inside the full suite (ordering check).
- Re-run twice back-to-back without teardown (env-leak check).
- Re-run with concurrency = N (data-race check).
You don’t need fancy ML to classify. The varied conditions surface the category on their own.
The Quiet Anti-Pattern: The Flake Registry As Hiding Place
A flake registry is good — a file or database where known flakes are logged. But it’s also where dead problems go to hide.
The discipline I enforce: every entry has three fields.
- Category (one of the four above).
- Owner (a person, not a team).
- Deadline (a date, not “soon”).
If you can’t fill those in, you don’t have a flake. You have a problem you haven’t looked at yet. Don’t dignify it with a registry entry; you’re lying to future-you.
I check the registry quarterly. Anything past deadline either gets fixed, gets escalated, or — sometimes — gets deleted because the underlying feature is gone and nobody noticed. All three outcomes are better than the status quo of letting it rot.
Three War Stories
The “flake” that was a real bug. A test had been on the quarantine list for six weeks. Nobody had touched it. I ran the harness: 1/5 pass. Definitely a flake. Then I read the failure. The test was hitting an endpoint that returned a 500 about 20% of the time under load. That’s not a flaky test. That’s a 20% error rate. The “flake” label had been hiding a real bug for a month and a half. We filed the ticket the same afternoon. The fix went out the door inside a week.
Lesson: when the harness says FLAKE, read what’s actually failing. The harness decides whether the test is consistent. Whether the thing being tested is flaky is still your job to investigate.
The “real bug” that was ordering. Opposite direction. A teammate was convinced a test had caught a regression. They’d run it locally, watched it fail, filed the ticket, gone home. I reran in isolation: 5/5 pass. Reran inside the suite: 1/5 pass. ORDERING. A test earlier in the suite was leaving a configuration toggle in the wrong state. The “regression” was a teardown gap that had existed for a month. Different conversation, different fix, no ticket needed against the product team.
The flake that took three rounds. And then the honest one. A test was failing in CI about 15% of the time. Five runs in isolation: 5/5 pass. Five runs in the full suite: 5/5 pass. Five runs with concurrency = 4: 3/5 pass. DATA_RACE. Three test threads were each trying to claim the same shared resource, and the test happened to be the one that lost the race most often. The harness needed three different shapes to surface the answer. It still found it in twenty minutes.
I tell these stories together because they’re the same lesson from three angles: the label “flaky” is not a diagnosis. It’s the absence of one. The harness exists to force you toward an actual diagnosis.
What This Isn’t
Two honest caveats.
It isn’t a fix. The harness diagnoses; you still have to do the engineering. It will tell you “this looks like ORDERING” — it won’t refactor your test setup for you.
And it isn’t a substitute for designing tests that don’t go flaky in the first place. Tests with explicit waits, owned preconditions, hermetic data, and concurrency-aware fixtures rarely make it onto the registry at all. The harness is a triage tool, not a hygiene one.
Steal This
The minimum viable version, in pseudocode:
def diagnose(test_id, n=5):
results_isolated = run_in_isolation(test_id, n)
pass_rate = passes(results_isolated) / n
if pass_rate == 1.0: return STABLE
if pass_rate == 0.0: return BROKEN
# It's a flake. Narrow the category.
results_suite = run_inside_suite(test_id, n)
results_repeat = run_back_to_back(test_id, n)
results_parallel = run_with_concurrency(test_id, n, threads=4)
hints = []
if worse(results_suite, results_isolated): hints.append("ORDERING")
if worse(tail(results_repeat), head(results_repeat)): hints.append("ENV_LEAK")
if worse(results_parallel, results_isolated): hints.append("DATA_RACE")
if not hints: hints.append("TIMING") # default
return FLAKE, hintsAbout thirty lines once you fill in the helpers. The intelligence is in what conditions you vary and what their comparison tells you. Everything else is plumbing.
Wire it to a one-line CLI command — flake-check <test_id> — and wire that command into your “we think this is flaky” pull-request label. Now nobody on the team gets to call a test flaky without thirty seconds of evidence.
You’d be amazed how much quieter the quarantine list gets!
