A flaky Playwright test is one that sometimes passes and sometimes fails without an intentional change to the test or application. When that happens, do not automatically add more retries.
Use this rule:
|
What is happening? |
What to do |
|---|---|
|
Temporary or intermittent failure |
Retry while investigating |
|
Known flaky test is disrupting the wider suite |
Quarantine temporarily |
|
Root cause is understood |
Fix it |
The goal is always to get back to a reliable test. Retries and quarantine are temporary ways to manage instability, not substitutes for fixing it.
Leapwork Play helps make recurring instability visible through the test execution history. In the demonstrated workflow, the Run log surfaces Flaky runs alongside passed, partial, and failed executions, helping you identify tests that need investigation instead of treating every successful rerun as a healthy test.
How should you handle a flaky Playwright test?
Start by establishing whether the failure is repeatable. If the test fails consistently under the same conditions, investigate and fix the underlying test or application problem. If it fails intermittently, use retries carefully while you investigate why the result changes between runs.
A simple decision flow is: A test that passes after retry can still be flaky. Passing eventually is not the same as being reliable.
How does Play help identify flaky tests?
In Leapwork Play, the Run log provides an execution history for the RunList, including passed, partial, and failed runs.
The demonstrated Play workflow also surfaces a Flaky count, making recurring instability visible across executions rather than requiring you to judge reliability from a single run.
Use that history to identify patterns:
Run repeatedly → compare outcomes → identify flaky behavior → investigate the test
This is particularly useful when a test passes on one run and fails on another: the successful run should not hide the instability.
When should you retry a flaky test?
Use retries when the failure appears intermittent and you still need the suite to continue while the cause is investigated.
In Playwright, retries can be configured, for example:
import { defineConfig } from '@playwright/test';
export default defineConfig({
retries: 2
});
A retry gives the test another attempt. It does not fix whatever caused the first attempt to fail.
If the pattern becomes:
Run 1 → Fail → Retry → Pass
Run 2 → Pass
Run 3 → Fail → Retry → Pass
treat the test as flaky and investigate it.
When should you quarantine a flaky test?
Quarantine a test when its known instability is affecting the usefulness of the wider test suite and you cannot fix it immediately.
Think of quarantine as: Remove from critical execution temporarily → investigate → fix → return to normal execution
Do not use quarantine simply because a test failed once. A quarantined test should also not become a forgotten test. Its underlying problem still needs to be resolved.
When should you fix the test?
Fix the test when you can identify the underlying cause. Common areas to investigate include unstable or changed locators, timing or synchronization problems, shared test data, authentication or session state, dependencies between tests, application defects, and environment-specific behaviour.
The fix should address why the result changes, rather than hiding the failure with additional retries.
How Play helps with flaky-test investigation
|
Playwright test behavior |
How Leapwork Play helps |
|---|---|
|
Test results change between executions |
Run history makes repeated outcomes visible |
|
A failed test passes on another run |
Flaky information helps surface recurring instability |
|
You need to understand which test is unreliable |
Inspect the RunList execution history and individual results |
|
Root cause needs investigation |
Open the relevant test and work from the recorded test steps and execution information |
Play complements Playwright by giving you a test execution workflow around the underlying Playwright tests, making it easier to see patterns across runs, not just the outcome of one execution.
Don't use retries to hide flakiness
If a test repeatedly needs retries to pass, increasing the retry count can make your pipeline look healthier without making the test more reliable. Use retries to keep transient failures manageable while you diagnose them not to redefine an unstable test as passing.
Best practices
-
Use Play's Run log and flaky-test information to identify recurring instability across executions.
-
Fix the underlying cause whenever it can be identified.
-
Use retries temporarily for intermittent failures while investigating them.
-
Quarantine only when a known flaky test is disrupting the wider suite.
-
Do not treat a test that passes after retry as automatically healthy.
-
Look at multiple executions before deciding whether a failure is genuinely flaky.
-
Return quarantined tests to normal execution after the problem is fixed.
Troubleshooting
|
Problem |
What to do |
|---|---|
|
Test fails every time |
Treat it as a reproducible failure and investigate the root cause |
|
Test sometimes passes and sometimes fails |
Compare executions and investigate the changing condition |
|
Test usually passes after retry |
Treat repeated retry-dependent success as evidence of flakiness |
|
One flaky test disrupts the wider suite |
Consider temporary quarantine while fixing it |
|
Retry count keeps increasing |
Stop masking the problem and investigate why the original attempt fails |
|
Play shows repeated flaky behavior |
Inspect the affected test and compare its execution history |
Frequently asked questions
What is a flaky Playwright test?
A flaky test produces inconsistent results without an intentional change to the test or application—for example, passing on one run and failing on another.
Should I retry a flaky Playwright test?
Retries can be useful temporarily for intermittent failures, but repeated reliance on retries is a reason to investigate the test.
When should I quarantine a flaky test?
Quarantine when a known flaky test is disrupting the wider suite and cannot be fixed immediately. Treat quarantine as temporary.
Is a test still flaky if it passes after a retry?
Yes. A successful retry does not remove the fact that the original execution failed unexpectedly.
How does Leapwork Play help identify flaky tests?
Play's Run log provides execution history and surfaces flaky behavior across runs, helping you identify tests whose outcomes are inconsistent rather than judging reliability from a single execution.
Should I fix, retry, or quarantine first?
Fix when the root cause is known. Retry temporarily when the failure is intermittent and under investigation. Quarantine when known instability is disrupting the wider suite and cannot be fixed immediately.