---
title: "Playwright Flaky Tests: Detect Them in CI with Retries"
description: "Find Playwright flaky tests in CI from retries, JSON reports and shard results, then rank them by flake rate, confirm with repeatEach and quarantine."
canonical: "https://currents.dev/posts/playwright-flaky-tests-detect-them-in-ci-with-retries"
updated: "2026-10-22"
---
# Playwright Flaky Tests: Detect Them in CI with Retries
Published 2026-10-22 by Currents Team.
Canonical URL: https://currents.dev/posts/playwright-flaky-tests-detect-them-in-ci-with-retries
Find Playwright flaky tests in CI from retries, JSON reports and shard results, then rank them by flake rate, confirm with repeatEach and quarantine.
---
A test goes red on main. Someone clicks re-run, the job goes green, and the pull request merges. A week later the same test fails on another branch, someone else clicks re-run, and nobody connects the two failures. Each re-run burns CI minutes and hides the pattern. It also teaches the team that a red build probably means nothing, which is how a real regression gets waved through.
That test is flaky: it fails intermittently. Timing-sensitive tests tend to flake on shared CI runners long before they flake on a laptop, so the person who wrote the test is often the last to see it fail.
Playwright already has the label you need. When [retries](https://playwright.dev/docs/test-retries) are on, Playwright marks a test flaky if it failed on the first run and passed on a retry. By default, though, failing tests aren't retried, so no test ever gets that label. Once they are, Playwright's [JSON reporter](https://playwright.dev/docs/test-reporters#json-reporter) produces an object with all information about the test run.
Turning retries on, saving that report from every run and a few CI steps are enough. A short script ranks your flaky tests by the runs they flaked in, and with [`failOnFlakyTests`](https://playwright.dev/docs/api/class-testconfig#test-config-fail-on-flaky-tests), the run exits with an error if any tests are marked as flaky.
## Retries are a flake detector, not a fix
By default, a test that fails and then passes on a retry is labeled flaky and does not fail the run. Playwright re-runs a failing test until it passes or reaches the retry limit, and the docs present retries as a way to deal with tests that fail now and then. For detection, the useful part of a retry is the label it produces.
With retries on, Playwright labels a test **flaky** when it failed on the first run and passed on a retry. It labels a test **failed** when it failed on the first run and on every retry. That split, an outcome read from [all of a test's attempts rather than any single one](/posts/test-status-translation-guide#attempts-vs-outcome), is the signal the rest of this pipeline collects.
The two labels need different work. A failed test failed every attempt, so it points at a real bug or a broken environment, and the next run will likely show it again. A flaky test passed at least once on the same code. A single result proves nothing about it, and you need its history before anyone can act.
None of this happens by default: a suite without a retry setting never labels a single test flaky.
A team turns on retries, the red builds stop, and the flaky label lands in reports nobody opens. The flaky tests are still in the suite. Each one now costs an extra attempt whenever it flakes, and a green build hides that cost.
Treat each flaky label as a bug report someone has to read. Count the labels per test across runs, and give the tests that flake most an owner.
### When not to retry
A test that fails on every attempt is not flaky. Playwright retries it until the maximum number of retries is reached, and after each failure it discards the worker process and browser and starts a new one. Fix that test, and don't raise the retry count to get it green.
## Turn on retries in CI only, and trace the first retry
Retries in the wrong place hide a flake from the one person best placed to fix it.
The config sets one retry count for every environment. With retries on, a test that fails on its first run but passes when retried is reported as flaky rather than as a failure, so the run stays green on the author's laptop.
```ts
// Anti-pattern: local runs retry too, so a flaky test passes there and is only reported as flaky
import { defineConfig } from '@playwright/test';
export default defineConfig({
retries: 3,
});
```
Follow the basic example on the [configuration page](https://playwright.dev/docs/test-configuration): retry only when `process.env.CI` is set. In the same file, set `trace: 'on-first-retry'`, which records and keeps a trace only for a test's first retry, so every flaky test leaves a trace behind for debugging.
```ts
import { defineConfig } from '@playwright/test';
export default defineConfig({
// retry in CI, keep local runs honest
retries: process.env.CI ? 2 : 0,
use: {
trace: 'on-first-retry',
},
});
```
`retries` is the maximum number of retry attempts a failed test gets. Two retries are enough to tell flaky from failed. Each extra retry adds time to every test that fails for real, and tells you nothing new.
### Change the count for one run
The [`--retries`](https://playwright.dev/docs/test-cli) CLI flag sets the maximum retry count for flaky tests, zero for no retries, and its default is no retries:
```bash
npx playwright test --retries=2
```
Use it for a one-off CI run, or set it to zero, for no retries, when every failure should stay a failure. The confirmation job further down does exactly that.
## Write the flaky outcome to a JSON report
On CI, Playwright's default reporter is `dot`, and the dot reporter has its own marker for a test that passed on a retry. That marker lives in a job log that scrolls away and that no script reads. You can use multiple reporters at the same time: for example, `list` for terminal output and `json` to get a JSON file with the test results. With the `json` reporter, the `PLAYWRIGHT_JSON_OUTPUT_NAME` environment variable writes the report to a file you can upload:
```bash
PLAYWRIGHT_JSON_OUTPUT_NAME=results.json npx playwright test --reporter=dot,json
```
If your dashboards read JUnit XML, check what they get: Playwright's JUnit report leaves retries out by default, so a test that passed on a retry looks like any other pass. A request to mark flaky tests in it for CI dashboards and history tracking was closed for limited engagement ([#35592](https://github.com/microsoft/playwright/issues/35592)). We'd take flakiness from Playwright's own test outcome instead.
A script walks every result, counts each failed attempt as a failure and each passed attempt as a pass. But `testResult.status` is the status of one test result, and `testCase.results` holds the results for each run of the test. Playwright calls a test flaky when it failed on the first run but passed when retried, so a script that counts attempts counts it as a failure and a pass at once. The flake rate is wrong before the first run is stored.
Classify each test with `testCase.outcome()`. The Reporter API gives it as the test's outcome, and the docs point out that it is not the same as `testResult.status`. Keep the attempts for context: `testResult.retry` gives each retry attempt a sequential number. You can create a custom reporter by implementing a class with some of the reporter methods. This one writes every test's outcome and attempts to `flake-summary.json` and prints the flaky ones:
```ts
// flake-summary.ts
import fs from 'node:fs';
import type { FullConfig, Reporter, Suite } from '@playwright/test/reporter';
class FlakeSummary implements Reporter {
private suite?: Suite;
onBegin(config: FullConfig, suite: Suite) {
this.suite = suite;
}
onEnd() {
const tests = (this.suite?.allTests() ?? []).map((test) => ({
// project, file and title: the key the history script groups by
id: test.titlePath().filter(Boolean).join(' > '),
outcome: test.outcome(),
attempts: test.results.map((result) => ({ retry: result.retry, status: result.status })),
}));
fs.writeFileSync('flake-summary.json', JSON.stringify(tests, null, 2));
for (const test of tests.filter((t) => t.outcome === 'flaky')) {
const attempts = test.attempts.map((a) => `#${a.retry} ${a.status}`).join(', ');
console.log(`flaky: ${test.id} (${test.attempts.length} attempts: ${attempts})`);
}
}
}
export default FlakeSummary;
```
Add it to the reporter list:
```bash
PLAYWRIGHT_JSON_OUTPUT_NAME=results.json npx playwright test --reporter=dot,json,./flake-summary.ts
```
### Check the setup with a test that flakes on purpose
Before trusting the pipeline, feed it a known flake. This test fails at random:
```ts
// tests/flaky-canary.spec.ts: delete once the pipeline is checked
import { test, expect } from '@playwright/test';
test('flaky canary', () => {
expect(Math.random()).toBeGreaterThan(0.5);
});
```
Run it with `CI` set, so the earlier `retries: process.env.CI ? 2 : 0` config turns retries on, a few times if needed, until one run shows the dot reporter's flaky marker. In that run the canary failed on the first run but passed when retried, which Playwright labels flaky. Its `testCase.outcome()` is then `flaky`. Then delete the test.
```bash
CI=1 PLAYWRIGHT_JSON_OUTPUT_NAME=results.json npx playwright test tests/flaky-canary.spec.ts --reporter=dot,json,./flake-summary.ts
```
## Merge shard blob reports so flaky results aren't lost
Sharding breaks the single-file picture. You shard a suite by passing `--shard=x/y`, and each shard has [its own test report](/posts/playwright-ci-at-scale-github-gitlab#artifact-handling-and-the-blob-reporter-pattern). A summary written on shard 1 only knows about the tests that ran on shard 1.
Each shard writes its own `results.json` and `flake-summary.json`, and the analysis reads whichever shard's file it downloads first. The flaky tests from every other shard never reach the count, and each test's flake rate comes out too low.
Have each shard write a blob report. A blob report holds every test that ran, its results and attachments such as traces, and blob reports can be merged and converted to any other Playwright report. Download every shard's blob into one directory such as `all-blob-reports`; blob report names contain the shard number, so they don't clash. Then merge them once, into one JSON report and one flake summary:
```yaml
name: e2e
on:
push:
branches: [main]
pull_request:
jobs:
test:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
shardIndex: [1, 2, 3, 4]
shardTotal: [4]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: lts/*
- run: npm ci
- run: npx playwright install --with-deps
- run: npx playwright test --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }} --reporter=dot,blob
- uses: actions/upload-artifact@v4
if: ${{ !cancelled() }}
with:
name: blob-report-${{ matrix.shardIndex }}
path: blob-report
retention-days: 1
merge:
if: ${{ !cancelled() }}
needs: test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: lts/*
- run: npm ci
- uses: actions/download-artifact@v4
with:
path: all-blob-reports
pattern: blob-report-*
merge-multiple: true
- run: npx playwright merge-reports --reporter=json,./flake-summary.ts ./all-blob-reports
env:
PLAYWRIGHT_JSON_OUTPUT_NAME: results.json
# one data point per run, named with the run ID
- uses: actions/upload-artifact@v4
with:
name: flake-history-${{ github.run_id }}
path: |
results.json
flake-summary.json
```
`npx playwright merge-reports` reads all blob reports from the passed directory and merges them into a single report. The `if: ${{ !cancelled() }}` condition is there so the merge job still runs when a shard fails. Don't drop it: red runs are the ones most likely to hold flaky tests.
## Keep every run's report and compute a flake rate
One run tells you which tests flaked today. Deciding what to fix takes knowing which tests flake week after week, and that takes history. The last step of the merge job above uses [`actions/upload-artifact`](https://github.com/actions/upload-artifact) with `name: flake-history-${{ github.run_id }}` and both `results.json` and `flake-summary.json` as its `path`, so every run becomes one data point.
A test's flake rate is the number of runs where its outcome was flaky, divided by the number of runs where it ran. Count failed runs in a separate column: a test that failed every attempt is broken, not flaky.
The script marks each test red or green per run and calls the red share the flake rate. A test that fails on every attempt tops the list at 100% though it isn't flaky, and a test that failed and then passed on a retry ends green and looks clean. Filtering by a failure-rate band doesn't help: it still mixes broken tests with flaky ones, and it never sees the flakes a retry turned green.
Count outcomes. Download the last 20 summaries from main with the GitHub CLI, then run the ranking script:
```bash
mkdir -p history
for id in $(gh run list --workflow e2e.yml --branch main --status completed --limit 20 --json databaseId --jq '.[].databaseId'); do
(mkdir -p "history/$id" && cd "history/$id" && gh run download "$id" -n "flake-history-$id") || true
done
# needs the tsx package installed: npm install --save-dev tsx
npx tsx flake-rate.ts history
```
```ts
// flake-rate.ts
import fs from 'node:fs';
import path from 'node:path';
type Entry = { id: string; outcome: string };
type Stats = { runs: number; flaky: number; failed: number };
const dir = process.argv[2] ?? 'history';
const stats = new Map();
for (const run of fs.readdirSync(dir)) {
const file = path.join(dir, run, 'flake-summary.json');
if (!fs.existsSync(file)) continue;
const entries: Entry[] = JSON.parse(fs.readFileSync(file, 'utf8'));
for (const { id, outcome } of entries) {
if (outcome === 'skipped') continue;
const s = stats.get(id) ?? { runs: 0, flaky: 0, failed: 0 };
s.runs += 1;
if (outcome === 'flaky') s.flaky += 1;
// outcome() reports a test that failed every attempt as 'unexpected'
if (outcome === 'unexpected') s.failed += 1;
stats.set(id, s);
}
}
const ranked = [...stats.entries()]
.filter(([, s]) => s.flaky > 0)
.sort(([, a], [, b]) => b.flaky / b.runs - a.flaky / a.runs);
for (const [id, s] of ranked.slice(0, 10)) {
const rate = Math.round((s.flaky / s.runs) * 100);
console.log(`${rate}% flaky (${s.flaky}/${s.runs} runs, ${s.failed} failed) ${id}`);
}
```
The script ignores skipped results, ranks every test that flaked at least once, highest rate first, and prints the top ten with the failed count beside the rate.
### Decide what to act on
Act on any test that was flaky in more than one of the last 20 runs on main. A single flaky run can be a bad runner. Two is a pattern. Post the top ten to the team channel once a week rather than sending an alert for every re-run: people read a weekly list, and they mute a stream of alerts.
If you'd rather not maintain the download-and-count scripts, [Currents Test Explorer](https://docs.currents.dev/dashboard/test-suite-performance-explorer/tests-explorer) shows a Flakiness Rate for each test across its recorded runs, computed as flaky results divided by the results you select.
## Confirm a suspect with repeatEach and retries off
A high flake rate makes a test a suspect, not a confirmed flake. Confirm it before someone spends a day on it.
Someone picks up the suspect, runs it once on a laptop, sees it pass and closes the ticket as cannot-reproduce.
```bash
# Anti-pattern: one run on fast hardware, and a retry can turn a failure into a pass
npx playwright test tests/checkout.spec.ts --retries=2
```
One pass on fast hardware says nothing about a test that fails some of the time. With retries on, a failed first attempt can still end in a pass, and `--fail-on-flaky-tests` is off by default, so a flaky result doesn't fail the run.
`testConfig.repeatEach` sets how many times each test repeats, and the docs describe it as useful for debugging flaky tests. Read it from an environment variable so normal runs still run each test once:
```ts
import { defineConfig } from '@playwright/test';
export default defineConfig({
retries: process.env.CI ? 2 : 0,
// 1 on normal runs; the confirmation job sets REPEAT_EACH
repeatEach: Number(process.env.REPEAT_EACH ?? 1),
use: {
trace: 'on-first-retry',
},
});
```
Then run the suspect in a manually triggered CI job with `--retries=0`, so no retry can hide a failure:
```yaml
name: confirm-flake
on:
workflow_dispatch:
inputs:
test_file:
description: Spec file to repeat, for example tests/checkout.spec.ts
required: true
jobs:
confirm:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: lts/*
- run: npm ci
- run: npx playwright install --with-deps
- run: npx playwright test "$TEST_FILE" --retries=0 --reporter=dot
env:
TEST_FILE: ${{ inputs.test_file }}
REPEAT_EACH: 50
```
Run the confirmation on a CI runner rather than a laptop. Timing flakes often need a slower, busier machine than the one on your desk, and the runner is where this test flaked.
### Read the result
If only a few repeats fail, the test is flaky. Quarantine it and look for the cause.
If every repeat fails, the test isn't flaky at all: treat it as a real bug or a broken environment.
A run with no failures doesn't clear the test either, because the sample may be too small. Raise `REPEAT_EACH` or run the job again before closing the ticket.
## Quarantine with a tag, then fail the build on new flakes
A confirmed flake that stays in the main job keeps costing retries, and nobody owns it. Move it out. To tag a test, pass a details object when you declare it or add an @-token to its title; tags must start with `@`.
```ts
import { test, expect } from '@playwright/test';
// Quarantined: owner and exit rule live in the tracking ticket
test('applies a discount code at checkout', { tag: '@quarantine' }, async ({ page }) => {
await page.goto('https://example.com/checkout');
await page.getByLabel('Discount code').fill('SPRING10');
await page.getByRole('button', { name: 'Apply' }).click();
await expect(page.getByText('Discount applied')).toBeVisible();
});
```
The main CI job then skips every test with that tag using `--grep-invert`:
```bash
npx playwright test --grep-invert @quarantine
```
Every quarantine tag needs an owner and an exit rule. Ours: the test comes back once the confirmation job runs it with no failures. Without both, quarantine becomes a list of tests nobody runs.
Tag-based quarantine has a cost: the test stops running, so it stops gathering evidence, and its flake rate freezes on the day you tagged it. Playwright has no marker that keeps a test running and counts its failures as flaky rather than failed: a maintainer turned that request down, pointing to `test.fail()` for a test that always fails ([#27653](https://github.com/microsoft/playwright/issues/27653)).
We recommend the tag for most flakes. The confirmation job is the exit rule, and it answers the one question you have while a fix is in progress: does the test still fail? A frozen flake rate costs nothing while you wait for that yes or no. Keep a quarantined test running only when you need its history to keep growing, for example when it flakes too rarely for one confirmation run to clear it. The [Currents quarantine action](https://docs.currents.dev/guides/currents-actions/reference/actions) does that: it runs the test but ignores its failures and reports its results with the status set to skipped. Errors in a `beforeAll` hook aren't covered by the action, and those tests still report as failed.
Someone discovers the flaky label and turns on `failOnFlakyTests` the same day. Every known flake now turns the build red at random, and the team switches the setting off within a week.
Quarantine the known flakes first, then turn the setting on for CI. `testConfig.failOnFlakyTests` makes the run exit with an error if any test is marked flaky, and the docs call it useful on CI. The CLI flag `--fail-on-flaky-tests` does the same for one run and is off by default. The config option shipped in Playwright 1.52, for a user who wanted it in the config file rather than only on the command line, so CI and local runs could differ ([#34397](https://github.com/microsoft/playwright/issues/34397)). Set it for CI only:
```ts
import { defineConfig } from '@playwright/test';
export default defineConfig({
retries: process.env.CI ? 2 : 0,
repeatEach: Number(process.env.REPEAT_EACH ?? 1),
// known flakes are quarantined, so any flake left is new
failOnFlakyTests: !!process.env.CI,
use: {
trace: 'on-first-retry',
},
});
```
From here on, a new flake fails the build on the run where it first appears.
## Read the cause from the flaky attempt's trace
The attempt that failed and the trace you kept are often not the same attempt. With `trace: 'on-first-retry'`, Playwright records and keeps a trace only for a test's first retry. When a flaky test passes on that first retry, the trace shows the attempt that passed. Read it next to the first attempt's result in `testCase.results` and look for where the two runs part ways.
The symptom narrows down the cause.
If the same step fails every time the test flakes, suspect a timing assumption: a fixed sleep, or an order of events the test takes for granted. Wait on the condition instead of the clock, assert on the outcome the user sees rather than an intermediate element state, and turn off animations where the app lets you.
If the test fails only in sharded or parallel CI runs, suspect a backend that slows down under load, or data that two workers share. Mock the slow endpoint or give that one step more time. Give each test its own data, created through the API before the test and deleted after it. Random values aren't isolation: two workers can draw the same one.
When an element is missing or wrong, suspect a DOM that changes with A/B flags or wrappers that render only sometimes. Locate by role, label or text instead of CSS class chains or XPath positions, and wait for a list to appear before picking an item from it.
The flake stops after someone raises the test timeout in the config. Every real hang now takes longer to fail, and the race is still there, waiting for a slower runner.
Leave the global timeout alone. Find the one slow step, then mock what it waits on or give that step alone more time. A [checklist for debugging Playwright timeouts](/posts/debugging-playwright-timeouts#the-debugging-checklist-for-playwright-timeouts) helps you find that step.
### Done means a flat line
Count a fix as done when the test's flake rate stays at zero over the next runs on main, not when one re-run passes. The history you're already collecting is the proof.
## Where this setup stops helping
The scripts are yours to maintain, with the same [ongoing cost as any custom reporter](/posts/playwright-custom-reporters-build-your-own#the-ongoing-cost-maintaining-a-custom-reporter). The summary keys each test by project, file and title, so renaming a test or moving it to another file starts its history from zero. A rate also means little until a test has many runs behind it: one flaky run out of three is noise.
The policy is yours too. `failOnFlakyTests` exits with an error if any test is marked flaky, so one flaky test is enough to fail the run. Playwright has no setting that tolerates a few: a request for a maximum number of flaky tests per run was closed for limited engagement ([#37002](https://github.com/microsoft/playwright/issues/37002)). We'd hold main at zero flakes outside quarantine anyway, and let the quarantine list be the buffer. Every flake gets a named owner the week it lands on the list. A quarantined test comes back when the confirmation job runs it with no failures. One that nobody is working on gets fixed or deleted, since a test that never runs protects nothing.
The re-run button never gave you any of that. The weekly list does, as long as every entry on it has a name next to it. A flake rate nobody owns only goes up.
## Frequently asked questions
### How does Playwright detect flaky tests?
With retries on, Playwright labels a test flaky when it failed on the first run and passed on a retry. It labels a test failed when it failed on the first run and on every retry. [Retries are a flake detector](#retries-are-a-flake-detector-not-a-fix) explains why the two labels need different work.
### Does Playwright retry failed tests by default?
No. The `--retries` CLI flag sets the maximum retry count for flaky tests, zero for no retries, and its default is no retries. Use it for a one-off CI run, as [changing the count for one run](#change-the-count-for-one-run) shows.
### Does a flaky test fail the Playwright run?
By default, a test that fails and then passes on a retry is labeled flaky and does not fail the run. `testConfig.failOnFlakyTests` makes the run exit with an error if any test is marked flaky, and the CLI flag `--fail-on-flaky-tests` does the same for one run and is off by default. [Quarantine the known flakes first](#anti-pattern-failing-on-flakes-before-quarantine-exists), then turn the setting on for CI.
### How do I find flaky tests when Playwright runs in shards?
Each shard has its own test report, so a summary written on shard 1 only knows about the tests that ran on shard 1. Have each shard write a blob report, then [merge the blob reports](#merge-shard-blob-reports-so-flaky-results-arent-lost) with `npx playwright merge-reports`, which reads all blob reports from the passed directory and merges them into a single report.
### How do I calculate a flake rate for Playwright tests?
A test's flake rate is the number of runs where its outcome was flaky, divided by the number of runs where it ran. Count failed runs in a separate column: a test that failed every attempt is broken, not flaky. The [flake rate section](#keep-every-runs-report-and-compute-a-flake-rate) has a script that ranks tests from each run's saved summary.
### How do I confirm a Playwright test is really flaky?
`testConfig.repeatEach` sets how many times each test repeats, and the docs describe it as useful for debugging flaky tests. Run the suspect in a manually triggered CI job with `--retries=0`, so no retry can hide a failure, as the [confirmation job](#confirm-a-suspect-with-repeateach-and-retries-off) does.
### How do I quarantine flaky tests in Playwright?
To tag a test, pass a details object when you declare it or add an @-token to its title; tags must start with `@`. The main CI job then skips every test with that tag using `--grep-invert`. Every [quarantine tag](#quarantine-with-a-tag-then-fail-the-build-on-new-flakes) needs an owner and an exit rule.