Flaky Test Cost: Calculate Wasted CI Spend in Playwright

A Playwright build can go green with a few tests marked flaky in the report. Each of those tests failed at least once before it passed, and every failed attempt ran on a CI runner you pay for. Nobody clicked rerun. The minutes are on the bill anyway.

If you price that from a spreadsheet of team guesses (reruns per engineer, a flake-rate percentage, an hourly rate, a runner price), you get a total nobody can check. Nothing in it points at a test you could fix.

Playwright already records a result for every run of each test, with that run's running time in milliseconds. So you don't have to estimate the compute side of flaky test cost. You can measure it from your own runs, and the measurement doubles as a list of tests to fix, most expensive first.

What cost calculators assume

A spreadsheet cost model is only as good as its inputs, and most of them are guesses: how often each engineer reruns a pipeline, how many minutes a rerun steals, what share of runs hits a flaky failure, what an engineer hour costs and what a runner minute costs.

The shape is fine. Compute, investigation time, context switching and deployment delay are real costs, and splitting them apart makes the total easier to argue about. The numbers are the weak part. Change one guess and the total moves with it, and the model can't tell you which guess was wrong.

Published figures don't fix the inputs. A flaky rate or an annual cost that another company or an industry survey reports describes that company's suite, its pipeline and its payroll, not yours. Use one as a sanity check on the total you compute, never as an input to it.

A model built on reruns also counts only what people do. A Playwright retry needs nobody: the test fails, runs again and passes, and Playwright doesn't fail the run over a flaky test unless you ask it to (the --fail-on-flaky-tests flag defaults to false). Those retry minutes never show up as a rerun, so a rerun-based model can't price them.

If you need a figure for engineer time today, keep a survey-based estimate for that side and label it as an estimate. Measure the compute side.

Retries versus manual reruns

Flaky tests cost a Playwright pipeline in two different ways, and each needs its own line in the calculation.

The first is retries. Playwright doesn't retry a failing test by default. With retries set, a failing test runs again until it passes or hits the maximum number of retries. The trace viewer docs use 2 retries on CI and 0 locally.

Retries also label every test. A test that failed its first run and passed on a retry is flaky. A test that failed its first run and every retry is failed. That label splits flaky from broken for each test.

Each retry costs more than the test's own runtime, too. When a test fails, Playwright discards the whole worker process and its browser and starts a new one, and with retries on, the new worker starts by retrying the failed test. Every retry pays for a fresh worker and browser first.

Broken tests pay the most, because they never stop at a pass. A user running three retries reported that a broken test's failure was output four times (#24587). The issue was closed as not planned.

The second cost is the manual rerun. Someone sees a red pipeline, repeats a whole job or shard by hand and waits for it to finish. That rerun bills compute for everything it repeats plus an engineer's time for the wait, so it can't share a line with retries.

// tests/checkout.spec.ts
import { test, expect, type Page } from '@playwright/test';

// Anti-pattern: one flaky test retries every test in the group
test.describe.configure({ mode: 'serial' });

let page: Page;

test.beforeAll(async ({ browser }) => {
  page = await browser.newPage();
});

test('add item to cart', async () => {
  await page.goto('/products/blue-mug');
  await page.getByRole('button', { name: 'Add to cart' }).click();
  await expect(page.getByTestId('cart-count')).toHaveText('1');
});

test('check out', async () => {
  await page.goto('/cart');
  await page.getByRole('button', { name: 'Check out' }).click();
  await expect(page).toHaveURL(/\/order\//);
});

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  use: { baseURL: process.env.BASE_URL },
});

// tests/checkout.spec.ts
import { test, expect } from '@playwright/test';

test('add item to cart', async ({ page }) => {
  await page.goto('/products/blue-mug');
  await page.getByRole('button', { name: 'Add to cart' }).click();
  await expect(page.getByTestId('cart-count')).toHaveText('1');
});

test('check out', async ({ page }) => {
  // Builds its own cart instead of relying on the test above
  await page.goto('/products/blue-mug');
  await page.getByRole('button', { name: 'Add to cart' }).click();
  await page.goto('/cart');
  await page.getByRole('button', { name: 'Check out' }).click();
  await expect(page).toHaveURL(/\/order\//);
});

Exporting the JSON report

GitHub keeps a workflow run's logs and artifacts for a set number of days, and artifacts are retained for 90 days by default. When running with --reporter=json, the PLAYWRIGHT_JSON_OUTPUT_NAME environment variable writes the report to a file. The JSON reporter produces an object with all information about the test run, so the test data the cost scripts need comes from that one object.

When you run with --reporter=json, set the PLAYWRIGHT_JSON_OUTPUT_NAME environment variable to write the report to a file. You don't have to give up readable CI output for it: Playwright can use several reporters at once, such as list for the terminal and json for the results file.

Inside the report, each test keeps one result per attempt (testCase.results in the reporter API). Each result carries its running time in milliseconds (duration), and when a test is retried multiple times, each retry attempt is given a sequential number (retry). Those fields, with each attempt's status, are what you need to price a failed attempt.

name: e2e
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: lts/*
      - run: npm ci
      - run: npx playwright install --with-deps
      - name: Run Playwright
        run: npx playwright test --reporter=list,json
        env:
          PLAYWRIGHT_JSON_OUTPUT_NAME: results.json
      - name: Keep the JSON report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: playwright-json-${{ github.run_id }}-${{ github.run_attempt }}
          path: results.json

Pricing retry minutes

Every attempt that didn't pass is waste. For a flaky test, that is every attempt before the final pass. For a failed test, it is every attempt.

Failed tests cost the most per test. With retries: 2, a test that fails every time fails its first run and both retry attempts, and none of those minutes buys anything.

Timeouts make it worse. Playwright enforces a per-test timeout, 30 seconds by default, that covers the test function, fixture setup and beforeEach hooks, so an attempt that hits it runs that whole time. A timed-out attempt is the most expensive retry you can buy.

A failed test doesn't stop the run either: Playwright continues in a new worker with the next test, so a run that is already red keeps spending. One user asked for a way to stop CI once a single test had failed every retry, saying the max-failures limit counted tests that had been retried successfully (#26350). The reporter replied that there is no way to exit Playwright at the correct moment with the existing features. The issue was closed as completed on 2024-09-11 (#26350).

To price the waste, convert the summed milliseconds to minutes and multiply by the per-minute rate on your own CI bill. Treat the result as a lower bound. duration is each attempt's running time, and every failure also costs Playwright a new worker and browser.

The scripts here read the report through one helper, which flattens it into one entry per test and project:

// read-report.ts
import { readFileSync } from 'node:fs';

export type Attempt = { status: string; duration: number; retry: number };
export type TestEntry = { id: string; status: string; results: Attempt[] };

type ReportTest = { projectName: string; status: string; results: Attempt[] };
type ReportSpec = { title: string; file: string; tests: ReportTest[] };
type ReportSuite = { specs?: ReportSpec[]; suites?: ReportSuite[] };

// An attempt that failed or timed out bought nothing, whatever the test's final outcome
export const isWasted = (attempt: Attempt) =>
  attempt.status === 'failed' || attempt.status === 'timedOut';

export function readReport(path: string): TestEntry[] {
  const report = JSON.parse(readFileSync(path, 'utf-8')) as { suites: ReportSuite[] };
  const entries: TestEntry[] = [];
  const walk = (suites: ReportSuite[]) => {
    for (const suite of suites) {
      for (const spec of suite.specs ?? []) {
        for (const test of spec.tests) {
          entries.push({
            id: `${spec.file} > ${spec.title} [${test.projectName}]`,
            status: test.status,
            results: test.results,
          });
        }
      }
      walk(suite.suites ?? []);
    }
  };
  walk(report.suites);
  return entries;
}

// flaky-only-cost.ts
import { readReport } from './read-report';

// Anti-pattern: prices only tests that ended flaky; a test that failed every retry never matches
const flaky = readReport(process.argv[2]).filter((test) => test.status === 'flaky');
const wastedMs = flaky.reduce(
  (sum, test) => sum + test.results.slice(0, -1).reduce((s, r) => s + r.duration, 0),
  0,
);
console.log(`Flaky retry time: ${(wastedMs / 60000).toFixed(1)} min`);

// retry-cost.ts
// Usage: npx tsx retry-cost.ts results.json <runner-rate-per-minute>
import { readReport, isWasted } from './read-report';

const [reportPath, rateArg] = process.argv.slice(2);
const rate = Number(rateArg);

const rows = readReport(reportPath)
  .map((test) => {
    const wasted = test.results.filter(isWasted);
    const minutes = wasted.reduce((sum, r) => sum + r.duration, 0) / 60000;
    return {
      test: test.id,
      outcome: test.status,
      wastedAttempts: wasted.map((r) => r.retry).join(','),
      minutes: Number(minutes.toFixed(2)),
      dollars: Number((minutes * rate).toFixed(2)),
    };
  })
  .filter((row) => row.wastedAttempts !== '')
  .sort((a, b) => b.minutes - a.minutes);

console.table(rows);
const total = rows.reduce((sum, row) => sum + row.minutes, 0);
console.log(`Retry waste: ${total.toFixed(1)} min, $${(total * rate).toFixed(2)}`);

Pricing manual CI reruns

Manual reruns are where compute and engineer time land on the same line. Count them from your CI's run history, not from a survey of how often people think they click rerun. A rerun count still misses failed runs nobody reran and every retry inside a green build, but the JSON report already covers the second.

Price each rerun in two parts. Compute is jobs repeated × minutes per job × your per-minute rate, and in a sharded run the jobs repeated depends on which rerun someone chose. Engineer cost is the pipeline's measured wall-clock time × the hourly rate of whoever waits on it. Measuring the wait replaces a guessed number of minutes per rerun and the velocity-loss estimates built on top of it.

// rerun-cost.ts
// Set the inputs from your CI bill and a recent pipeline run
type RerunInput = {
  jobsRepeated: number;
  minutesPerJob: number;
  runnerRatePerMinute: number;
  waitMinutes: number;
  hourlyRate: number;
};

export function rerunCost(input: RerunInput): number {
  const compute = input.jobsRepeated * input.minutesPerJob * input.runnerRatePerMinute;
  const waiting = (input.waitMinutes / 60) * input.hourlyRate;
  return compute + waiting;
}

const env = (name: string) => Number(process.env[name]);
const shared = {
  minutesPerJob: env('MINUTES_PER_JOB'),
  runnerRatePerMinute: env('RUNNER_RATE_PER_MINUTE'),
  waitMinutes: env('PIPELINE_WALL_CLOCK_MINUTES'),
  hourlyRate: env('ENGINEER_HOURLY_RATE'),
};

const allJobs = rerunCost({ ...shared, jobsRepeated: env('SHARDS') });
const failedJobs = rerunCost({ ...shared, jobsRepeated: env('FAILED_SHARDS') });

console.log(`Re-run every job:   $${allJobs.toFixed(2)}`);
console.log(`Re-run failed jobs: $${failedJobs.toFixed(2)}`);

How sharding multiplies cost

In a CI pipeline, each shard can run as a separate job, using the pipeline's own hardware. On GitHub-hosted runners, compute is billed based on the total processing time used by each runner type, so re-running all jobs of an N-shard workflow bills the runner time of all N shard jobs again.

Retries behave differently. After a failure, Playwright Test starts a new worker process, and with retries enabled that worker starts by retrying the failed test and continues from there. We'd expect retry spend to grow with the number of flaky and failing tests. On GitHub Actions, if a workflow that normally takes 10 minutes fails after 5 minutes and is re-run successfully, the total cost is based on 15 minutes.

Suite growth makes the rerun line worse. More tests mean more pipelines contain at least one flaky failure, even if no single test gets flakier, and each of those pipelines is a candidate for a full rerun.

Once you shard, rerunning every job is the biggest compute line you control. Fix how people rerun before you tune retries.

Ranking the costliest tests

A total tells you how much flakiness costs. A ranking tells you what to fix. Merge the JSON reports from weeks of CI runs (the flaky-test guide covers keeping every run's report and computing a flake rate) and rank tests by wasted minutes, not by flake rate. A few weeks of real history also replaces a baseline built by running the suite over and over on purpose.

Frequency matters as much as flakiness. A flakiness impact score, flakiness rate × executions, shows why: a slightly flaky test that runs often can matter more than a very flaky test that rarely runs. A duration impact column, duration × executions, adds the cumulative time spent running a test.

Put a dollar figure on each row. The per-test figure is the honest return on fixing that test: what it cost over the period you measured.

// rank-flaky-cost.ts
// Usage: npx tsx rank-flaky-cost.ts reports/ <runner-rate-per-minute>
import { readdirSync } from 'node:fs';
import { join } from 'node:path';
// read-report.ts is the helper shown in "Pricing retry minutes"
import { readReport, isWasted } from './read-report';

type Stats = { runs: number; flaky: number; totalMs: number; wastedMs: number };

const [dir, rateArg] = process.argv.slice(2);
const rate = Number(rateArg);
const byTest = new Map<string, Stats>();

for (const file of readdirSync(dir).filter((f) => f.endsWith('.json'))) {
  for (const test of readReport(join(dir, file))) {
    if (test.status === 'skipped') continue;
    const s = byTest.get(test.id) ?? { runs: 0, flaky: 0, totalMs: 0, wastedMs: 0 };
    s.runs += 1;
    if (test.status === 'flaky') s.flaky += 1;
    for (const result of test.results) {
      s.totalMs += result.duration;
      if (isWasted(result)) s.wastedMs += result.duration;
    }
    byTest.set(test.id, s);
  }
}

const rows = [...byTest].map(([test, s]) => {
  const flakyRate = s.flaky / s.runs;
  return {
    test,
    runs: s.runs,
    flakyRate: Number(flakyRate.toFixed(3)),
    flakinessImpact: Number((flakyRate * s.runs).toFixed(1)),
    durationImpactMin: Number((s.totalMs / 60000).toFixed(1)),
    wastedDollars: Number(((s.wastedMs / 60000) * rate).toFixed(2)),
  };
});

rows.sort((a, b) => b.wastedDollars - a.wastedDollars);
console.table(rows.slice(0, 20));

Past a few hundred runs

Across hundreds of runs, branches and shards, the script stops being enough. Currents' Test Explorer computes Flakiness Impact and Duration Impact from your recorded runs, and you can sort tests by either column.

Cutting the cost

Retries hide flaky test cost; they don't remove it. Keep them on CI so builds stay green, and price them from the report every week so the spend stays visible.

Where one file needs different handling, set its retries with test.describe.configure() instead of changing the global setting. Cap retries for a slow file whose failures tend to time out.

failOnFlakyTests in the config, or --fail-on-flaky-tests on the command line, makes the run fail when any test is flaky, and the flag is off by default. Turn it on only after you've fixed the top of the ranked list. Turned on early, it converts cheap retry minutes into red builds and expensive manual reruns.

This config shows the end state, and the test file caps retries for one slow file:

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  // Turn on once the top of the cost ranking is fixed
  failOnFlakyTests: !!process.env.CI,
  use: {
    baseURL: process.env.BASE_URL,
    trace: 'on-first-retry',
  },
});

// tests/reports-export.spec.ts
import { test, expect } from '@playwright/test';

// Slow file whose failures tend to time out: cap its retries instead of raising them suite-wide
test.describe.configure({ retries: 1 });

test('exports the monthly report', async ({ page }) => {
  await page.goto('/reports');
  const download = page.waitForEvent('download');
  await page.getByRole('button', { name: 'Export CSV' }).click();
  expect((await download).suggestedFilename()).toContain('.csv');
});

Fix the top-ranked tests rather than only quarantining them. A quarantine list stops the noise, but nobody learns why the test fails. The evidence is already there: with retries at 2 on CI, the trace viewer docs' setup records a trace on the first retry of a failed test.

In that trace, look for timing first (a fixed wait that is too short, a race with the app), then data or environment shared with other tests, then selectors that match the wrong element. Comparing the failed attempt with the passing retry is the fastest way to see which one it is. Adding a sleep turns a flaky failure into a slower pass. Make the fixed test isolated, with no state or data shared with other tests, so it retries alone and never drags a serial group with it.

For the manual-rerun line, Currents' Re-run Only Failed Tests reruns only the failed Playwright tests, including in sharded parallel runs, instead of repeating the whole suite.

Where the number stops helping

The measured figure is a floor. It leaves out root-cause debugging, real failures dismissed as flakes, lost trust in CI and deploys held up behind a red build.

Engineer time will usually outweigh compute, so put a developer-time estimate next to the measured lines. Show it across a range of loaded costs, and state its assumption: the lost time would otherwise have shipped product. A reader who sees the assumption won't mistake the estimate for a measurement.

Impact scores have a limit too. They're only meaningful for comparing tests with each other, not as dollar figures on their own. And if your suite is small and the retry minutes come to almost nothing, skip the calculator. Fix the one test everyone already reruns.

Everywhere else, you now have two measured lines (retry minutes from the JSON report, reruns from CI history) and one labeled estimate for engineer time. That is a figure you can defend in a planning meeting, and the ranked list beside it says which test to fix first.

Frequently asked questions