---
title: "Flaky Test Report: Build One Leadership Will Act On"
description: "Build a flaky test report from Playwright's retries and JSON reports: a reproducible flake rate, CI and engineer time lost, and a weekly trend."
canonical: "https://currents.dev/posts/flaky-test-report-build-one-leadership-will-act-on"
updated: "2026-11-02"
---
# Flaky Test Report: Build One Leadership Will Act On
Published 2026-11-02 by Walter Galvão.
Canonical URL: https://currents.dev/posts/flaky-test-report-build-one-leadership-will-act-on
Build a flaky test report from Playwright's retries and JSON reports: a reproducible flake rate, CI and engineer time lost, and a weekly trend.
---
> **TL;DR**
>
> A flaky test report is a recurring summary of how often tests change outcome without a code change, what that costs in time and CI spend, and what the team is doing about it. With retries on, Playwright marks a test flaky when the test failed on its first run and passed on a retry.
>
> 1. Set `retries` on CI and add the `json` reporter next to `list`.
> 2. Have each shard write a blob report, merge the blobs with `npx playwright merge-reports`, and archive one merged JSON file per pipeline run.
> 3. Add a small custom reporter that writes one row per test per run, keyed on `testCase.id`.
> 4. Compute flake rate per test and for the whole suite, and publish the window and the denominator next to the number.
> 5. Compare the current window with the previous one, and put the delta next to every figure.
Every week someone says the tests are flaky, and every week the flake work loses the planning meeting to a feature. It loses because "flaky" is a complaint with no number behind it. Nobody in the room can say how flaky, what it costs, or whether last month's fixes changed anything.
The usual evidence doesn't help. A single run's HTML report or CI log says what broke once. It can't say how often a test flakes or which way the trend is going.
Playwright already records the signal. With retries on, it marks a test flaky when the test failed on its first run and passed on a retry. On GitHub Actions, you keep it past the job by uploading it as an [artifact](https://docs.github.com/en/actions/concepts/workflows-and-actions/workflow-artifacts), which lets you persist data after a job has completed. If you keep it, you can walk into the meeting with three or four figures anyone can reproduce and a weekly trend, both built from Playwright's own run data, plus a list of what's quarantined and what's fixed.
## What leadership acts on
Leadership funds what it can price. A flaky test report is a recurring summary of how often tests change outcome without a code change, what that costs in time and CI spend, and what the team is doing about it. Each part answers something a VP or CTO decides on: money, risk and action.
Lead with four figures, each with its change from last week:
- Suite flake rate
- CI minutes lost to retries and reruns
- Engineer hours lost
- Quarantined versus fixed tests
Change the vocabulary too. "Flaky test" means little outside engineering, while "unreliable quality gate" means something to anyone who ships. "Retries" becomes "CI spend and merge delay." Name the cost leadership can't see on a dashboard: engineers stop trusting a red build, and nobody can say which tests are to blame.
Every figure needs a published formula and archived inputs. A number nobody can reproduce gets argued about instead of funded.
When leadership asks about flakiness, the easy answer is a pasted stack trace or a link to the latest HTML report. The [HTML report](https://playwright.dev/docs/test-reporters#html-reporter) covers a single test run. One run can't show a trend. Don't send leadership the HTML report. Someone asked for graphs of pass, fail, skip and flaky tests in the HTML report ([#29552](https://github.com/microsoft/playwright/issues/29552)). A maintainer folded the request into another issue.
Send figures with a window, a formula and a trend, built from archived run data instead of one run's output.
## How Playwright marks a test flaky
No retries, no flaky outcome. Playwright doesn't retry failing tests by default, and when [retries](https://playwright.dev/docs/test-retries) are on, it reruns a failing test until it passes or reaches the maximum number of retries. With retries at zero, a test that would have passed on a second try goes into the data as a plain failure.
A config that never sets `retries` runs every test once on CI. Every flake lands in the failed column next to the real regressions, and the report can't tell them apart.
```ts
// playwright.config.ts
import { defineConfig } from '@playwright/test';
// Anti-pattern: retries are never set, so CI records every flake as a failure
// and the flaky outcome never appears in the data.
export default defineConfig({
testDir: './tests',
reporter: 'list',
});
```
Set `retries` on CI and add the `json` reporter next to `list`. Playwright runs both at once: `list` keeps the terminal readable, and `json` leaves a results file behind.
```ts
// playwright.config.ts
import { defineConfig } from '@playwright/test';
export default defineConfig({
testDir: './tests',
// Retry on CI only; locally a failure should stop you right away.
retries: process.env.CI ? 2 : 0,
reporter: [['list'], ['json', { outputFile: 'results.json' }]],
});
```
For a one-off run, the `--retries` flag sets the same maximum from the command line:
```bash
npx playwright test --retries=2
```
### Flaky versus failed
With retries on, failing tests split in two. A test that failed on its first run and passed on a retry is flaky. A test that failed on its first run and on every retry is failed. The reporter API carries the same split: a test that passes on a second retry has the outcome `'flaky'`, so the flaky count is part of Playwright's own data.
Inside a test, `testInfo.retry` is zero on the first run, one on the first retry, and so on. Use it to [collect extra diagnostics only on retries](/posts/playwright-flaky-tests-detect-them-in-ci-with-retries#turn-on-retries-in-ci-only-and-trace-the-first-retry), so the evidence is there when the test reaches the top of the report.
Treat retries as diagnosis, not closure. A flaky pass still contains a real failure on its first attempt. The report has to count that failure, not accept the green build.
Retries catch only one kind of flakiness: a test that flips within a single run. If a test fails every attempt on one runner image or one branch, Playwright marks it failed there. Only grouping results across many runs shows that the same test passes everywhere else.
## Collecting results across shards
A rate needs a denominator, and one run doesn't give you much of one. The report needs every test from every shard, for every pipeline run, stored somewhere that outlives the CI job.
The JSON reporter produces an object with all the information about a run. When you run with `--reporter=json`, the `PLAYWRIGHT_JSON_OUTPUT_NAME` environment variable writes that object to a file.
In a sharded pipeline, the easiest results to grab are one shard's output or the HTML report from the last run. One shard counts only part of the suite. The last run has no history, so there's nothing to compute a rate from.
Have each shard write a blob report. Blob reports contain all the details of a run, can produce any other report later, and exist mainly to merge results from [sharded tests](https://playwright.dev/docs/test-sharding). Blob file names contain the shard number, so they don't clash in one directory, and `npx playwright merge-reports` reads every blob in a directory and merges them into a single report.
This GitHub Actions workflow runs four shards, merges their blobs into one JSON file per pipeline run, runs the history reporter from the next section, and uploads all of it:
```yaml
name: Playwright
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
shardIndex: [1, 2, 3, 4]
shardTotal: [4]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 22
- run: npm ci
- run: npx playwright install --with-deps
- run: npx playwright test --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }} --reporter=blob
- if: ${{ !cancelled() }}
uses: actions/upload-artifact@v4
with:
name: blob-report-${{ matrix.shardIndex }}
path: blob-report
merge:
if: ${{ !cancelled() }}
needs: test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 22
- run: npm ci
- uses: actions/download-artifact@v4
with:
path: all-blob-reports
pattern: blob-report-*
merge-multiple: true
- name: Merge shards into one JSON file
run: npx playwright merge-reports --reporter=json ./all-blob-reports
env:
PLAYWRIGHT_JSON_OUTPUT_NAME: results-${{ github.run_id }}.json
- name: Append history rows
run: npx playwright merge-reports --reporter=./reporters/history-reporter.ts ./all-blob-reports
- uses: actions/upload-artifact@v4
with:
name: flaky-report-data-${{ github.run_id }}
path: |
results-${{ github.run_id }}.json
history/
all-blob-reports/
```
Archive that JSON file for every pipeline run, tagged with commit, branch and pipeline ID. Each file describes a single run, so the archive is the report's history. Copy it to long-term storage if your CI artifacts expire before your reporting window ends.
Keep the blobs too. If you change what you measure next quarter, the old blobs can still produce any report, and the archive gives you one place to review errors across many runs instead of opening CI logs one at a time.
## A reporter that keeps history
The merged JSON file is complete, but it's awkward to query across hundreds of runs. A [small custom reporter](/posts/playwright-custom-reporters-build-your-own) writes exactly the rows the report needs. A [custom reporter](https://playwright.dev/docs/test-reporters) is a class that implements some of the reporter methods. Two are enough here. `onTestEnd` is called after a test finishes in the worker process, and `onEnd` is called after all tests have run or testing was interrupted.
Write one row per test per run: `testCase.id`, the outcome, how many results the test has, and each result's `testResult.retry` and `testResult.duration`. `testCase.results` holds a result for each run of the test. Each retry attempt gets a sequential number in `testResult.retry`, and `testResult.duration` is in milliseconds.
```ts
// reporters/history-reporter.ts
import fs from 'fs';
import path from 'path';
import type { Reporter, TestCase } from '@playwright/test/reporter';
const historyDir = process.env.HISTORY_DIR ?? 'history';
class HistoryReporter implements Reporter {
private tests = new Map();
onTestEnd(test: TestCase) {
// Keep the latest TestCase per ID; rows are written once the run is over.
this.tests.set(test.id, test);
}
onEnd() {
const run = {
runId: process.env.GITHUB_RUN_ID ?? `local-${Date.now()}`,
workflow: process.env.GITHUB_WORKFLOW ?? 'local',
branch: process.env.GITHUB_REF_NAME ?? 'local',
commit: process.env.GITHUB_SHA ?? 'local',
finishedAt: new Date().toISOString(),
};
let flaky = 0;
let failed = 0;
const rows = [...this.tests.values()].map((test) => {
const outcome = test.outcome();
if (outcome === 'flaky') flaky++;
if (outcome === 'unexpected') failed++;
return JSON.stringify({
...run,
testId: test.id,
title: test.title,
file: path.relative(process.cwd(), test.location.file),
outcome,
attempts: test.results.length,
results: test.results.map((r) => ({ retry: r.retry, durationMs: r.duration, status: r.status })),
});
});
fs.mkdirSync(historyDir, { recursive: true });
fs.appendFileSync(path.join(historyDir, 'tests.ndjson'), rows.map((row) => `${row}\n`).join(''));
fs.appendFileSync(
path.join(historyDir, 'runs.ndjson'),
`${JSON.stringify({ ...run, tests: rows.length, flaky, failed })}\n`,
);
}
}
export default HistoryReporter;
```
The reporter writes rows in `onEnd`, after the last retry, so each row carries the test's final outcome. Each GitHub-hosted runner except single-CPU ones is a new virtual machine, so a merge job there starts with an empty `history/` directory and its files hold that run alone. The scripts below read a `history/tests.ndjson` that you build by concatenating the archived files.
`runs.ndjson` gets one row per pipeline run with its flaky and failed counts, which lets you flag the runs that had flakes and filter on them. `tests.ndjson` is a per-test, per-run table, the same shape as a daily totals table.
Grouping rows by title looks fine until two files both have a test called "loads the dashboard", or one test runs in a Chromium project and a Firefox project. Their histories merge, and the per-test numbers no longer match anything anyone can rerun.
Key on `testCase.id`. Playwright computes it from the test file name, the test title and the project name, and the ID is unique within a Playwright session, so the same test in two projects gets two IDs. Because the title is one of the inputs, renaming a test starts a new history. Store the title next to the ID so you can stitch a renamed test back together by hand.
## Flake rate per test and suite
With the history rows in place, the first figure is simple division. Flake rate is flaky executions divided by executions over a stated window, times 100. Compute it per test and for the whole suite, and publish the window and the denominator next to the number. We count an execution as one test in one pipeline run that wasn't skipped, so a test retried twice still counts once.
Report failure rate separately. A test that failed on its first run and on every retry is failed, not flaky. Folding failures into the flake figure inflates it and hides real breakage.
### The per-test table
This script reads the history rows for a window, counts total, passed, failed, flaky and skipped results per test, keeps tests that flaked at least once and ran a minimum number of times, and ranks them:
```ts
// scripts/flake-table.ts
import fs from 'fs';
type Row = {
testId: string;
title: string;
file: string;
branch: string;
workflow: string;
finishedAt: string;
outcome: 'expected' | 'unexpected' | 'flaky' | 'skipped';
};
const windowDays = Number(process.env.WINDOW_DAYS ?? 28);
const minExecutions = Number(process.env.MIN_EXECUTIONS ?? 20);
const since = Date.now() - windowDays * 24 * 60 * 60 * 1000;
const rows: Row[] = fs
.readFileSync('history/tests.ndjson', 'utf-8')
.split('\n')
.filter(Boolean)
.map((line) => JSON.parse(line) as Row)
.filter((row) => Date.parse(row.finishedAt) >= since);
const pct = (part: number, whole: number) => (whole ? ((100 * part) / whole).toFixed(2) : '0.00');
type Stats = { title: string; file: string; total: number; passed: number; failed: number; flaky: number; skipped: number; lastFlaky: string };
const byTest = new Map();
for (const row of rows) {
const s = byTest.get(row.testId) ??
{ title: row.title, file: row.file, total: 0, passed: 0, failed: 0, flaky: 0, skipped: 0, lastFlaky: '' };
s.total++;
if (row.outcome === 'expected') s.passed++;
if (row.outcome === 'unexpected') s.failed++;
if (row.outcome === 'skipped') s.skipped++;
if (row.outcome === 'flaky') {
s.flaky++;
if (row.finishedAt > s.lastFlaky) s.lastFlaky = row.finishedAt;
}
byTest.set(row.testId, s);
}
const ranked = [...byTest.values()]
.map((s) => ({ ...s, executed: s.total - s.skipped }))
.filter((s) => s.flaky > 0 && s.executed >= minExecutions)
// Flaky executions (rate × executions) first, then the most recent flake.
.sort((a, b) => b.flaky - a.flaky || b.lastFlaky.localeCompare(a.lastFlaky));
console.table(ranked.map((s) => ({
test: s.title,
file: s.file,
executed: s.executed,
passed: s.passed,
failed: s.failed,
flaky: s.flaky,
flakeRate: pct(s.flaky, s.executed),
failureRate: pct(s.failed, s.executed),
lastFlaky: s.lastFlaky.slice(0, 10),
})));
const executed = rows.filter((r) => r.outcome !== 'skipped');
const suiteFlaky = executed.filter((r) => r.outcome === 'flaky').length;
console.log(`Suite flake rate, last ${windowDays} days: ${pct(suiteFlaky, executed.length)}% of ${executed.length} executions`);
function rateBy(key: (row: Row) => string) {
const groups = new Map();
for (const row of executed) {
const g = groups.get(key(row)) ?? { flaky: 0, executed: 0 };
g.executed++;
if (row.outcome === 'flaky') g.flaky++;
groups.set(key(row), g);
}
return [...groups].map(([group, g]) => ({ group, executed: g.executed, flakeRate: pct(g.flaky, g.executed) }));
}
console.table(rateBy((row) => row.branch));
console.table(rateBy((row) => row.file));
console.table(rateBy((row) => row.workflow));
console.table(rateBy((row) => String(new Date(row.finishedAt).getUTCHours())));
```
The last four tables group flake rate by branch, file, workflow and hour of day. A test that flakes mostly on one branch or at one hour points at that branch or at the environment, not at random timing.
Sorting the table by flake rate puts rarely run tests on top. A test that ran three times and flaked once shows 33%. A test that flaked 40 times in 2,000 runs shows 2%, with 40 times as many flaky runs behind it.
Rank by flaky executions, which is flake rate times executions and works out to the raw flaky count, and break ties on the last flaky date. Require a minimum number of executions before a test can enter the ranking. The script does both, so the list starts with the tests that flaked most often. Product risk is still your call: at the same count, a flaky checkout test outranks a flaky settings page. [Currents Test Explorer](https://docs.currents.dev/dashboard/test-suite-performance-explorer/tests-explorer) computes Flakiness Impact (flakiness rate × executions) and lets you choose which outcomes count toward the flakiness rate. Its % Flaky Trend shows how a test's flakiness changed between two equal time periods.
## CI minutes and engineer hours lost
A flake rate says how often. The next two figures say what it costs, and Playwright records each test result's running time in milliseconds, so retry minutes can be computed from its results. The [first run of a test](https://playwright.dev/docs/api/class-testinfo#test-info-retry) has `testInfo.retry` equal to zero and the first retry has it equal to one, so add up the `testResult.duration` of the retry results to get retry time summed across workers.
A failure also restarts the worker. When any test fails, Playwright discards the whole worker process and its browser, then continues from the next test in a new worker process. With retries enabled, the new worker process starts by retrying the failed test and continues from there.
Whole-pipeline reruns come from your CI system: GitHub Actions, for example, lets you re-run a workflow run, all failed jobs in it, or specific jobs. Count reruns of pipelines that failed on flaky tests and multiply by the job's minutes; that is [how flakiness shows up on the CI bill](/posts/github-actions-cost-how-to-cut-playwright-ci-minutes-without-losing-coverage#fix-5-treat-flakiness-as-a-billing-problem-not-just-a-reliability-problem).
Engineer time is an estimate, so show the formula:
```plaintext
engineer hours = flaky failures investigated × minutes per investigation ÷ 60
engineer cost = engineer hours × fully loaded hourly rate
```
Take the hourly rate from finance, and label minutes per investigation as an estimate wherever it appears. This script sums retry minutes per week and combines them with the CI and estimated inputs:
```ts
// scripts/flake-cost.ts
import fs from 'fs';
type Row = { finishedAt: string; results: { retry: number; durationMs: number }[] };
// Inputs from outside Playwright. Investigation counts and minutes are estimates: label them.
const reruns = Number(process.env.FLAKY_RERUNS ?? 0); // pipeline reruns on flaky failures, from CI
const jobMinutes = Number(process.env.JOB_MINUTES ?? 0); // minutes per pipeline run, from CI billing
const investigations = Number(process.env.INVESTIGATIONS ?? 0);
const minutesPerInvestigation = Number(process.env.MINUTES_PER_INVESTIGATION ?? 0);
const hourlyRate = Number(process.env.LOADED_HOURLY_RATE ?? 0); // from finance
function weekOf(iso: string): string {
const d = new Date(iso);
d.setUTCDate(d.getUTCDate() - ((d.getUTCDay() + 6) % 7)); // back to Monday
return d.toISOString().slice(0, 10);
}
const retryMs = new Map();
for (const line of fs.readFileSync('history/tests.ndjson', 'utf-8').split('\n').filter(Boolean)) {
const row = JSON.parse(line) as Row;
const ms = row.results.filter((r) => r.retry > 0).reduce((sum, r) => sum + r.durationMs, 0);
const week = weekOf(row.finishedAt);
retryMs.set(week, (retryMs.get(week) ?? 0) + ms);
}
for (const [week, ms] of [...retryMs].sort(([a], [b]) => a.localeCompare(b))) {
console.log(`Week of ${week}: at least ${(ms / 60000).toFixed(0)} retry minutes`);
}
const engineerHours = (investigations * minutesPerInvestigation) / 60;
console.log({
rerunMinutes: reruns * jobMinutes,
engineerHours,
engineerCost: engineerHours * hourlyRate,
});
```
### A worked month
The inputs below are placeholders that show the arithmetic, not benchmarks. Replace every one with yours.
| Input | Placeholder | Where it comes from |
| --- | --- | --- |
| Retry minutes (floor) | 1,200 | Retry durations in the history rows |
| Pipeline reruns on flaky failures | 60 | CI system |
| Minutes per pipeline run | 25 | CI billing |
| Rerun minutes | 1,500 | 60 × 25 |
| Flaky failures investigated | 45 | Team estimate |
| Minutes per investigation | 30 | Team estimate |
| Engineer hours | 22.5 | 45 × 30 ÷ 60 |
With those inputs, the month spent at least 1,200 minutes of test time on retries and 1,500 CI minutes on reruns, plus an estimated 22.5 engineer hours, priced at finance's rate. Leave release-delay cost out unless you can name the release that was blocked.
Some costs don't fit a formula. A flaky pass can hide a real regression, and a red build nobody trusts gets rerun instead of read. Name those in the headline, and don't put a price on them.
A cost slide built on an industry statistic or a round hourly rate looks authoritative until someone asks where the number came from. The answer is somebody else's blog post, and the rest of the report loses credibility with it.
Use your own retry minutes, your CI system's rerun count and finance's loaded rate. A smaller number you can trace beats a bigger one you can't.
## The one-page weekly report
If the report runs past one page, nobody reads the second. The layout:
1. A one-line headline that says which way things moved.
2. Three or four figures with week-over-week deltas.
3. The top 10 flaky tests with owner and last flaky date.
4. The quarantined and fixed lists.
5. One ask, such as a sprint for the checkout suite.
Skip a composite 0-100 health score unless you publish its formula and weights. A score that blends flake rate, failure rate and retry minutes invites the question of how, and without an answer nobody can defend it. Show the inputs instead.
Track flaky test count, suite flake rate, quarantine age and time to fix, with at least eight weeks of trend, so the report shows whether a fix worked or just coincided with a quiet week. Take the owner column from [CODEOWNERS](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners), the file GitHub uses to define the individuals or teams that are responsible for code in a repository.
The quick version of the report is this week's numbers and nothing else. A flake rate with no prior period has no direction. Without a trend, nobody can tell whether the money spent on fixes worked.
Compare the current window with the previous one, and put the delta next to every figure. This script puts a delta next to the flake rate and the flaky-test count, labels each top-10 test as new or existing, counts resolved tests, and renders the page as Markdown:
```ts
// scripts/weekly-report.ts
import fs from 'fs';
type Row = { testId: string; title: string; file: string; finishedAt: string; outcome: string };
type Entry = { title: string; owner: string; enteredAt: string; expiresAt: string; fixedAt: string | null };
const DAY = 24 * 60 * 60 * 1000;
const now = Date.now();
const rows: Row[] = fs.readFileSync('history/tests.ndjson', 'utf-8').split('\n').filter(Boolean).map((l) => JSON.parse(l));
const owners: Record = JSON.parse(fs.readFileSync('owners.json', 'utf-8'));
const registry: Entry[] = JSON.parse(fs.readFileSync('quarantine.json', 'utf-8'));
const ownerOf = (file: string) => Object.entries(owners).filter(([prefix]) => file.startsWith(prefix)).sort((a, b) => b[0].length - a[0].length)[0]?.[1] ?? 'unowned';
function windowStats(start: number, end: number) {
const executed = rows.filter((r) => {
const t = Date.parse(r.finishedAt);
return t >= start && t < end && r.outcome !== 'skipped';
});
const flaky = executed.filter((r) => r.outcome === 'flaky');
return {
flaky,
flakyIds: new Set(flaky.map((r) => r.testId)),
flakeRate: executed.length ? (100 * flaky.length) / executed.length : 0,
};
}
const cur = windowStats(now - 7 * DAY, now);
const prev = windowStats(now - 14 * DAY, now - 7 * DAY);
const delta = (a: number, b: number) => `${a >= b ? '+' : ''}${(a - b).toFixed(1)}`;
const resolved = [...prev.flakyIds].filter((id) => !cur.flakyIds.has(id)).length;
const top = new Map();
for (const row of cur.flaky) {
const t = top.get(row.testId) ?? { row, count: 0, last: '' };
t.count++;
if (row.finishedAt > t.last) t.last = row.finishedAt;
top.set(row.testId, t);
}
const top10 = [...top.values()].sort((a, b) => b.count - a.count || b.last.localeCompare(a.last)).slice(0, 10);
const quarantined = registry.filter((e) => !e.fixedAt);
const fixed = registry.filter((e) => e.fixedAt && Date.parse(e.fixedAt) >= now - 7 * DAY);
const age = (iso: string) => Math.floor((now - Date.parse(iso)) / DAY);
const lines = [
`# Flaky tests, week ending ${new Date(now).toISOString().slice(0, 10)}`,
'',
`Suite flake rate ${cur.flakeRate.toFixed(2)}% (${delta(cur.flakeRate, prev.flakeRate)} pts). ` +
`${cur.flakyIds.size} flaky tests (${delta(cur.flakyIds.size, prev.flakyIds.size)}), ${resolved} resolved.`,
'',
'| Test | Owner | Flaky runs | Last flaky | Status |',
'| --- | --- | --- | --- | --- |',
...top10.map(({ row, count, last }) =>
`| ${row.title} | ${ownerOf(row.file)} | ${count} | ${last.slice(0, 10)} | ${prev.flakyIds.has(row.testId) ? 'existing' : 'new'} |`),
'',
'## Quarantined',
...quarantined.map((e) => `- ${e.title} (${e.owner}): ${age(e.enteredAt)} days, expires ${e.expiresAt}`),
'',
'## Fixed this week',
...fixed.map((e) => `- ${e.title} (${e.owner})`),
];
fs.writeFileSync('flaky-report.md', `${lines.join('\n')}\n`);
```
`owners.json` maps path prefixes to owners; you can build it from the directory entries in CODEOWNERS, whose patterns follow most of the same rules as gitignore files. `quarantine.json` is the registry from the next section. Add retry minutes from the cost script as the third figure.
## Quarantined and fixed lists
Quarantine is where measurement usually dies. Before a test goes in, reproduce the flake with `--repeat-each`, which runs each test N times, check the test's history, and assign an owner. If the fix is cheap, fix it now. Otherwise file a ticket with a target sprint.
The natural next step would be a marker that keeps a quarantined test running without failing on it. A user asked for exactly that ([#27653](https://github.com/microsoft/playwright/issues/27653)). A maintainer replied that there is already an option for always-failing tests and that a flaky marker had been considered and rejected; the request was closed with no plans to implement it.
The quickest quarantine is `test.fixme` and a comment. [`test.fixme()`](https://playwright.dev/docs/test-annotations) marks a test as failing, and Playwright doesn't run it, as opposed to the fail annotation.
```ts
// tests/checkout/coupon.spec.ts
import { test, expect } from '@playwright/test';
// Anti-pattern: fixme stops the test from running, so it leaves the data,
// and nobody owns bringing it back. "Flaky, look at it later."
test.fixme('checkout applies a saved coupon', async ({ page }) => {
await page.goto('/checkout'); // assumes use.baseURL in playwright.config.ts
await page.getByLabel('Coupon').fill('SPRING10');
await page.getByRole('button', { name: 'Apply' }).click();
await expect(page.getByText('10% off')).toBeVisible();
});
```
The test now produces no data. It drops out of the denominator, so the flake rate improves while coverage quietly shrinks, and nothing can show a fix or a regression.
Keep the test running and record the quarantine in a registry the report script reads: reason, owner, first-seen date, entry date, expiry and ticket.
```json
[
{
"testId": "7c41d2f09a8e3b6c5d1e-4f2a9b8c7d6e5f4a3b2c",
"title": "checkout applies a saved coupon",
"file": "tests/checkout/coupon.spec.ts",
"owner": "@payments-team",
"reason": "Coupon API times out when shards run in parallel",
"firstSeen": "2026-09-02",
"enteredAt": "2026-09-09",
"expiresAt": "2026-10-07",
"ticket": "PAY-1432",
"fixedAt": null
}
]
```
Each week the report labels flake as new (no flaky runs in the previous window), existing (flaky in both windows) or resolved (flaky last window, none in this one), and shows each quarantine's age against its expiry. An expired entry is a meeting item, not a silent extension.
### Leaving quarantine
A test moves to the fixed list only when it passes a repeated run with retries off:
```bash
npx playwright test tests/checkout/coupon.spec.ts --repeat-each=50 --retries=0
```
With zero retries, a failing attempt counts as a failure instead of a flaky pass. After that run, the test has to stay at zero flakes over the next window, with enough executions to mean something. Only then do you set `fixedAt` in the registry.
## Where the numbers mislead
A flake rate is only as honest as the setup behind it, and serial groups are the first place to look. In serial mode, a failing test causes the tests after it to be skipped, and all tests in the group are retried together. In one report on Playwright 1.39, a test in a serial group with retries failed every attempt it ran, yet the tests were identified as flaky and the CI passed ([#28322](https://github.com/microsoft/playwright/issues/28322)). Other users reported tests marked flaky rather than failed with retries above 0, and the issue was closed as completed on 2024-04-25. Check flaky counts for serial groups by hand before they go in the report.
Someone raises `retries` to calm a noisy week. Tests that failed every attempt under the old limit can now pass on a later retry, so failures turn into flakes, and lowering the limit does the reverse. The trend line now shows the config change, not the suite.
Keep `retries` constant across the reporting window. If you have to change it, start a new window on the day of the change and mark it on the trend.
### Gates and limits
`failOnFlakyTests` makes the run exit with an error if any test is marked flaky, and the [docs](https://playwright.dev/docs/api/class-testconfig) call it useful on CI. Turn it on as a gate only once quarantine exists and the flake rate is low. Turned on earlier, it makes every flaky test fail the run. The team learns to rerun instead of read. Retrying until green isn't a fix either: it hides the number this report exists to show.
A [self-built report stops scaling](/posts/playwright-test-observability-run-history-flakes-traces#when-diy-stops-paying-off) at some point. Renames are one sign: Playwright computes `testCase.id` from the test file name, test title and project name, so history keyed on it starts over when any of them is renamed.
## Automating the weekly send
A report someone rebuilds by hand every Friday stops going out the first busy week. Run the aggregation and report scripts on a weekly schedule against the archive of merged reports, and have the job email or post `flaky-report.md`.
Add a nightly scheduled run with `--repeat-each` on last week's top suspects. `--repeat-each ` runs each test N times (default: 1). Keep retries enabled on that run: tests that failed on the first run but passed when retried are marked flaky. Merge its blobs the same way as any pipeline run, and tag those runs so the report can show them apart from PR traffic.
When the archive, the scripts and the email become a job of their own, [Currents Automated Reports](https://docs.currents.dev/dashboard/automated-reports) can be scheduled weekly, daily or every 3 days. The report compares metrics from the selected period against a configurable look-back period. For spec files, the report calculates the flakiness rate as flaky executions divided by overall executions, where a flaky execution has at least 1 test detected as flaky. Projects with the Slack App connected at the organization level can deliver the same scheduled report to a Slack channel.
## What to ship first
You don't need all of this before the first report. Ship it in this order:
1. Week one: turn on retries on CI and archive one merged JSON file per pipeline run. Without history, nothing else works.
2. Week two: add the history reporter and the flake rate table, and share the top 10 with the team before leadership sees it. Owners catch the table's mistakes before a VP does.
3. After three or four weeks of data: send the first report with flake rate and retry minutes. Add engineer hours once the team agrees on the estimated inputs.
Every number on that first page will trace back to an archived run and a published formula. That changes the planning meeting: the argument moves from whether the numbers are real to which tests to fix first.
## Frequently asked questions
### How does Playwright detect flaky tests?
With retries on, Playwright marks a test flaky when the test failed on its first run and passed on a retry. Playwright doesn't retry failing tests by default, so with retries at zero, a test that would have passed on a second try goes into the data as a plain failure. [How Playwright marks a test flaky](#how-playwright-marks-a-test-flaky) shows the config that fixes this.
### How do I track flaky tests across sharded Playwright runs?
Have each shard write a blob report, then run `npx playwright merge-reports`, which reads every blob in a directory and merges them into a single report. Archive the merged JSON file for every pipeline run, tagged with commit, branch and pipeline ID, as [Collecting results across shards](#collecting-results-across-shards) shows. Each file describes a single run, so the archive is the report's history.
### How do I set up the Playwright JSON reporter?
Add the `json` reporter next to `list` in `playwright.config.ts`, with an `outputFile` such as `results.json`. Playwright runs both at once: `list` keeps the terminal readable, and `json` leaves a results file behind.
### How do I measure CI minutes lost to flaky tests?
Add up the `testResult.duration` of the retry results to get retry time summed across workers. Count reruns of pipelines that failed on flaky tests and multiply by the job's minutes. [CI minutes and engineer hours lost](#ci-minutes-and-engineer-hours-lost) includes a script and a worked month.
### Should I quarantine flaky Playwright tests with test.fixme?
No. Keep the test running and record the quarantine in a registry the report script reads: reason, owner, first-seen date, entry date, expiry and ticket.
### Should I turn on failOnFlakyTests in CI?
Turn it on as a gate only once quarantine exists and the flake rate is low. `failOnFlakyTests` makes the run exit with an error if any test is marked flaky, so turning it on earlier makes every flaky test fail the run. [Gates and limits](#gates-and-limits) covers the trade-off.
---
## About the author
**Walter Galvão**, Software Engineer. Walter Galvão is a software engineer at Currents. He writes about Playwright in practice: debugging flaky tests, structuring suites for CI, and the engineering workflows that keep large test suites reliable.
LinkedIn: https://www.linkedin.com/in/waltergalvao/