A flaky test produces different outcomes without a relevant change to the code under test. In continuous integration, that uncertainty wastes reruns, interrupts delivery, and trains engineers to distrust red builds. A flaky test quarantine can restore a usable signal by removing a confirmed unstable check from merge blocking while it continues to run in an observable lane. The quarantine is a temporary incident-control state, not a verdict that the test no longer matters.
The dangerous version of quarantine is a skip annotation with no owner or deadline. It makes the pipeline green by discarding evidence and allows product defects, infrastructure faults, and broken test assumptions to age together. A sound policy defines entry proof, release impact, retry meaning, ownership, expiry, repair priority, and an objective recovery gate. Every quarantined test must remain visible as reliability debt.
Distinguish flakiness from a reproducible failure
Do not quarantine merely because a test failed once. First rerun the same revision with the same inputs and preserve the original artifact. If it fails consistently, treat it as a normal product, test, dependency, or environment defect. If outcomes alternate, classify the axis that changes: order, worker count, timing, locale, time zone, browser, seed, network, database state, feature flag, or external dependency. Classification determines both containment and repair.
Retries are useful evidence because frameworks can identify a test that failed initially and passed later. Playwright, for example, categorizes such a case as flaky and by default does not retry failed tests unless configured. But a passing retry does not invalidate the first failure. It increases evidence of nondeterminism. Record every attempt, worker replacement, duration, trace, seed, and environment rather than reporting only the final green status.
| Observed pattern | Initial classification | Containment | Investigation focus |
|---|---|---|---|
| Fails on every rerun | Reproducible defect | Block according to test criticality | Product behavior, fixture, dependency, or environment |
| Fails then passes unchanged | Probable flake | Apply bounded retry and open evidence record | Timing, state leak, race, or nondeterministic data |
| Fails only after another test | Order-dependent flake | Isolate worker or suite while repairing | Shared global, database, cache, or cleanup |
| Fails only on one host class | Environment-sensitive | Remove unhealthy host pool if broad | Image, font, clock, CPU, kernel, or browser |
| Fails against external service | Dependency uncertainty | Use contract boundary and explicit service policy | Rate limit, sandbox stability, network, or credentials |
Require evidence before quarantine entry
An entry record should include stable test identity, owning team, first and recent failure revisions, attempt outcomes, environment, failure signature, affected product capability, criticality, linked issue, proposed expiry, and containment method. Automate collection from the test reporter. GitLab's documented quarantine metadata links a quarantined test to an issue and can classify reasons such as flaky behavior, a product bug, environment failure, dependency, or investigation. The general principle is traceability, even when another runner is used.
Set a confidence rule appropriate to run frequency. A high-volume test may need two inconsistent outcomes on the same revision; a rare, expensive journey may need a focused diagnostic run. Do not demand hundreds of repetitions before containing a test that blocks every team, but never erase the triggering failure. If evidence suggests a product race rather than test-only instability, keep a separate product incident and decide release risk explicitly.
Choose the narrowest containment mechanism
Prefer moving the individual test to a nonblocking quarantine job that still runs on relevant changes. Skipping it entirely is a last resort for a test that damages shared state, invokes unsafe external actions, or consumes disproportionate resources. Quarantining a whole file or suite hides healthy checks and expands blind spots. The marker should be machine-readable so CI can reject missing issue links, owners, or expiration dates.
Keep release-critical assertions represented. If the flaky test is the only automated control for authentication, money movement, destructive operations, consent, or data loss, quarantine cannot silently remove that gate. Add a temporary deterministic check at a lower layer, require manual release evidence, disable the affected change path, or accept risk through the named release authority. Pipeline convenience does not decide customer exposure.
Give retries explicit semantics
Use a small retry count and create a clean execution boundary between attempts. Playwright discards the failed worker process and starts a new worker, which helps isolate contamination, but hooks run again and server-side state may persist. A retry must either use an idempotent fixture or perform verified cleanup. Repeating a payment, email, account deletion, or shared mutation can turn a test reliability feature into a production-like side effect.
Store attempt number and outcome as dimensions. Useful metrics include first-attempt pass rate, retry recovery rate, persistent failure rate, quarantine count, age, and runtime tax. A suite with 99 percent final success after retries can still be operationally poor if many first attempts fail. Alert when a retry surge clusters by host, dependency, or test setup because the correct action may be draining infrastructure rather than editing dozens of tests.
| Metric | Definition | Decision it supports | Common trap |
|---|---|---|---|
| First-attempt pass rate | Tests passing before any retry | Health of the original CI signal | Publishing only final success |
| Retry recovery rate | Initial failures that pass within the retry bound | Probable nondeterminism and retry cost | Calling recovered tests healthy |
| Quarantine age | Time since entry for each active item | Escalation and stale debt control | Using an average that hides old tests |
| Quarantine coverage loss | Critical behaviors without a blocking substitute | Release-risk decision | Counting tests rather than behaviors |
| Failure concentration | Share by host, signature, owner, or dependency | Systemic repair priority | Assigning each symptom independently |
Assign owners, service levels, and expiry
The code-owning team owns behavioral intent; a test-platform team owns runner and infrastructure faults. When classification is uncertain, assign one temporary directly responsible owner until handoff is accepted. Prioritize by coverage loss, failure frequency, affected teams, customer risk, and repair effort. A low-frequency test guarding financial reconciliation may outrank a noisy cosmetic assertion even if the latter causes more reruns.
Set expiry at entry. On expiry, CI should escalate or restore blocking according to policy, not silently extend the date. Extensions require fresh evidence, an approver, revised mitigation, and a new deadline. Establish a small quarantine budget by team or capability; crossing it triggers engineering leadership review because accumulated uncertainty can no longer be managed test by test.
Diagnose from controlled experiments
Reduce the test to a repeatable experiment. Fix the random seed, freeze or inject time, vary worker count, repeat in isolation and after suspected predecessors, and compare clean versus reused state. Capture traces, network records, console output, server logs, database identifiers, and resource pressure. Avoid adding arbitrary sleeps: they can move the race window while extending every run. Wait for an observable condition with a bounded timeout instead.
Common repairs include unique test data, transactional fixture reset, per-test browser or process isolation, deterministic clocks, explicit event completion, stable selectors, controlled service virtualization, and removal of shared mutable globals. Verify cleanup itself. A teardown that returns success before asynchronous deletion finishes can contaminate the next test. Where eventual consistency is part of the product contract, poll the authoritative state with a deadline and useful failure output.
Use a proof-based recovery gate
A merged fix does not end quarantine. Run the repaired test repeatedly across the environments and concurrency modes that previously exposed the problem. The sample size should reflect its historical rate; a test that failed one in five runs needs fewer clean executions than one that failed one in a thousand. Also rerun likely neighboring tests to ensure the repair did not serialize the suite or hide shared-state leakage.
Recovery evidence includes the fix revision, causal explanation, before and after attempt data, clean runs, restored blocking lane, and removal of temporary substitutes. Continue heightened monitoring for a defined observation period. If the signature returns, reopen the same reliability record so recurrence is visible. Creating a fresh issue every time understates age and loses the history needed to identify an incomplete fix.
Review the quarantine as a reliability portfolio
Hold a short recurring review with quality platform and owning teams. Examine oldest items, critical coverage gaps, recurring signatures, infrastructure clusters, expired entries, and retry tax. The meeting should unblock repairs and assign systemic work, not debate every passing test. Publish trends by capability and owner while avoiding individual scorecards that incentivize deleting difficult tests.
Audit policy bypasses: disabled quarantine jobs, broad skip patterns, issue links that no longer resolve, ownership changes, and tests renamed without carrying metadata. Preserve test identity independently from display names when possible. A repository check can compare quarantine markers with the registry and fail changes that add untracked skips or remove a test without resolving its open record.
Example: isolate an order-dependent test
A customer-search test fails only after an account-deletion test on parallel CI. Reruns in a new worker pass. The entry evidence shows both tests share a fixed tenant and the deletion job completes asynchronously. The team quarantines only the search test in a visible lane, adds a temporary API-level customer lookup gate, assigns the customer-platform owner, and sets a seven-day expiry.
The repair creates a unique tenant per test and waits on the deletion job's authoritative completion event before teardown succeeds. Recovery runs vary order and worker count, then restore the search test to blocking. The temporary API check is removed after observation. This outcome fixes data isolation and teardown semantics; simply raising the retry count would have preserved the race and multiplied deletion traffic.
Key takeaways
- Quarantine only after preserving evidence that distinguishes nondeterminism from a reproducible defect.
- Keep quarantined tests running and visible whenever they are safe to execute.
- Report first-attempt outcomes even when retries recover the pipeline.
- Protect release-critical behavior with a deterministic substitute or explicit risk decision.
- Require owner, issue, reason, mitigation, and expiry for every entry.
- Remove quarantine only after causal repair and environment-relevant clean runs.
Frequently asked questions
Should quarantined tests be skipped in CI?
Prefer a nonblocking quarantine lane. Skip only when execution is unsafe, destructive, or prohibitively expensive, and record a substitute control plus expiry. A skipped test produces no evidence that its failure rate changed.
How many clean runs prove a fix?
Use the historical failure rate and the exposing conditions. There is no universal number. Include order, concurrency, host, browser, and timing variations relevant to the cause, then monitor after blocking is restored.
Can a product bug be quarantined as a flaky test?
The check may be contained to keep CI usable, but the product defect remains a separate release risk. Do not classify a real race or intermittent service failure as test-only merely because it is hard to reproduce.
Conclusion
A flaky test quarantine protects delivery only when it preserves accountability. Evidence-based entry, narrow containment, bounded retries, critical-behavior substitutes, explicit ownership, forced expiry, and proof-based recovery keep uncertainty temporary. The aim is not a cosmetically green pipeline; it is a CI system whose failures engineers can trust and whose blind spots remain visible until repaired.