TL;DR

This article covers how to quarantine flaky Playwright tests so that a known flake stops blocking merges without hiding a real regression. It uses plain Playwright and CI code: quarantine annotations, a custom reporter that judges each quarantined test after the run, sharded runs, and a nightly check that decides when a test leaves quarantine.

Keep each quarantined test in the normal run with an owner, an issue and an expiry date, and still fail the build when its failure looks real. A known flake should pass, while a new error in the same test or a failure on every attempt should fail the run.

  1. Before you tag a test, measure it: turn on retries in CI only, then run the suspect test with repeatEach, first alone and then with the rest of its file.
  2. Register a quarantine reporter that never turns a failed run green. It fails the run when a quarantined test fails every attempt, throws an error its signature doesn't match, has a broken record or pushes the count past the cap, and when a test outside quarantine is flaky.
  3. Leave failOnFlakyTests off in the blocking run, and in sharded jobs check the cap once, over the merged report.

A flaky checkout test blocks a merge. Someone hits re-run, it goes green, and the PR lands. After a month of that, the team re-runs every red build by reflex, and a red build no longer means something is broken. The first real regression gets re-run along with the noise.

Quarantine is supposed to stop that. It is the step between detecting a flaky test and fixing it, once you've done what you can to avoid flakes in the first place. Any suite of real size needs it: small per-test failure rates add up across hundreds of tests until a run with no flake at all is the exception.

The usual quarantine makes things worse, only more quietly. A test gets parked with test.fixme(), which marks it as failing, so Playwright won't run it. Or it gets a tag, and the blocking job skips the tests with that tag through --grep-invert. Or it moves to a second job that isn't allowed to fail the pipeline. The first two take the test out of the run. The third runs it into a report nobody opens. Either way, a regression in the code that test covers ships without anyone seeing it.

What works is a quarantine that keeps the test in the normal run, gives it an owner, an issue and an expiry date, and still fails the build when its failure looks real.

Quarantine is not a skip

A quarantined test keeps running and keeps reporting. It only stops gating merges. Playwright does not run a skipped test, and a test that fails and then passes on retry is reported as "flaky".

Anti-pattern

Fixme or grep-invert quarantine

One flaky test gets parked with test.fixme(), and another gets a tag so the blocking job can skip the tests with that tag. test.fixme() marks a test as failing, and Playwright doesn't run it. The docs show --grep-invert as the way to skip tests with a given tag, so the tagged test leaves the run as well.

// Anti-pattern: neither test runs in the blocking job, so neither can show a regression or a recovery
import { test, expect } from '@playwright/test';

test.fixme('applies a discount code', async ({ page }) => {
  await page.goto('https://example.com/cart');
  await page.getByLabel('Discount code').fill('SPRING');
  await page.getByRole('button', { name: 'Apply' }).click();
  await expect(page.getByText('Discount applied')).toBeVisible();
});

test('submits an order', { tag: '@quarantine' }, async ({ page }) => {
  await page.goto('https://example.com/checkout');
  await page.getByRole('button', { name: 'Place order' }).click();
  await expect(page.getByText('Order confirmed')).toBeVisible();
});
# The blocking job drops every @quarantine test before it runs
npx playwright test --grep-invert @quarantine

Neither test runs: Playwright will not run a test.fixme() test, and --grep-invert only runs tests that do not match. You can't watch either one recover, and a regression in the cart or checkout code goes out green.

Fix

Keep both tests in the normal run, tagged, and decide what their results mean after the run ends, in a reporter. A separate lane that runs only the tagged tests beats dropping them, but it helps only if someone reads its results.

The options side by side

ApproachDoes the test run?What you learn
test.fixme()NoNothing: Playwright does not run it
Tag plus --grep-invertNot in the blocking jobNothing from the blocking job
test.fail()YesA complaint if the test does not fail
Retry until greenYesA "flaky" label, which fails the run if failOnFlakyTests is on
QuarantineYesEvery result, without gating merges

test.fail() looks closer. Playwright runs the test, checks that it fails, and complains if it passes. A flaky test that passes most of the time would make Playwright complain on most runs, so test.fail() fits a test you expect to fail every time.

A second, non-blocking CI job for the tagged tests does keep their results. We'd rather keep one run: a second job pays for its own browser install and setup, and nobody has a reason to open its report.

You'll build the single-run version yourself. Someone asked for a way to keep tagged flaky tests running while their failures leave the exit code alone (#34679). A Playwright contributor replied that they strongly recommend fixing flaky tests instead of skipping them. The issue was closed in February 2025.

Confirming a real flake

Quarantine a test you haven't checked and you may park a regression, and once it carries the tag nobody looks at it as one. If you don't know yet which tests flake, find them from retries and JSON reports before you pick candidates.

Retries first

Playwright can't call anything flaky until retries are on. Retries are off by default; once you set retries, Playwright re-runs a failing test until it passes or reaches the maximum number of retries. Turn them on in CI only:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  // Retries only in CI: locally, a failure should stay a failure
  retries: process.env.CI ? 2 : 0,
  // 1 unless REPEAT_EACH is set, for measuring a suspect test
  repeatEach: Number(process.env.REPEAT_EACH ?? 1),
});

The docs call repeatEach useful for debugging flaky tests, and the measurement below uses it.

Playwright labels a test that failed on its first run and passed on a retry "flaky", and one that failed on its first run and every retry "failed". A flaky result on one commit is evidence that the test is intermittent. It is not the root cause, and it doesn't prove the failure isn't a regression.

Anti-pattern

Quarantine after one red run

A test fails once, someone adds the tag, and the PR merges. If the test depends on what ran before it, or the red run was a real regression that happened to pass its retry, it now sits in quarantine looking like every other flake.

Fix

Measure before you tag. When a test fails, Playwright discards the whole worker process and its browser and starts a new one, so the retry runs in a fresh process. On a retry, a new worker process starts with the failed test, so the tests that ran before it in the first worker don't run before it again. repeatEach sets how many times each test is repeated, so run the suspect test alone with it, then with the rest of its file. --test-list takes a file that lists tests in a format similar to the --list output, so put the suspect test's line in suspect.txt first:

# The suspect test alone, repeated
REPEAT_EACH=50 npx playwright test --test-list suspect.txt
# The same test with the rest of its file
REPEAT_EACH=50 npx playwright test tests/checkout.spec.ts

Without CI set, retries stay at 0, so every repetition reports a plain pass or fail. Failures when the test runs alone mean it is intermittent on its own. A test that is clean alone but red with its file points to order dependence or leaked state, which needs a fix in the tests, not a quarantine.

Rule out the rest

Before a test gets the tag, check the causes that aren't flakes. Run it on the commit before the change: if it passes there and fails every time here, you have a regression. A third-party service that went down is an outage, so mock the service instead of quarantining the test. Timing races, state shared between tests and differences in the CI environment are flake causes you can often fix on the spot, for example by resetting server-side state through your API in a beforeEach hook. Quarantine only a failure that keeps coming back after those checks.

Retries stay a stopgap, never a fix. Any test, hook or fixture can read testInfo.retry to see whether it is running as a retry. Use it to capture extra state on retry attempts (a server log, a dump of the test data) and give the owner something to debug.

Quarantine annotations

A tag tells the reporter a test is quarantined. It says nothing about who owns the test, which issue tracks it or when the quarantine ends, so it stays forever.

Anti-pattern

A bare quarantine tag

A one-word tag is the fastest way to get a red build out of the way:

// Anti-pattern: a bare tag records no owner, no issue and no end date
import { test, expect } from '@playwright/test';

test('submits an order', { tag: '@quarantine' }, async ({ page }) => {
  await page.goto('https://example.com/checkout');
  await page.getByRole('button', { name: 'Place order' }).click();
  await expect(page.getByText('Order confirmed')).toBeVisible();
});

Six months later nobody knows who added the tag or whether the bug behind it was fixed.

Fix

Keep the tag in the test's details object, since Playwright can filter tests by their tags during test execution, and add annotations for the owner, the issue, the expiry date and the known failure signature. The tag selects the test and the annotations describe it, the same split between tags and annotations that keeps a large suite organized:

import { test, expect } from '@playwright/test';

test(
  'submits an order',
  {
    tag: '@quarantine',
    annotation: [
      { type: 'quarantine-owner', description: 'payments-team' },
      { type: 'quarantine-issue', description: 'https://github.com/acme/shop/issues/482' },
      { type: 'quarantine-expires', description: '2026-12-15' },
      // A fragment of the error that only the flaky runs show
      { type: 'quarantine-signature', description: '<error text recorded from the flaky runs>' },
    ],
  },
  async ({ page }) => {
    await page.goto('https://example.com/checkout');
    await page.getByRole('button', { name: 'Place order' }).click();
    await expect(page.getByText('Order confirmed')).toBeVisible();
  },
);

Annotations have a type and a description and are available in the reporter API, so the whole quarantine record lives next to the test. A separate YAML manifest drifts away from the tests it describes, while annotations change in the same commit as the test. Point quarantine-issue at the ticket, so whoever owns the ticket owns the quarantine. For quarantine-signature, pick a fragment of the error message that only this flake produces. Don't start the types with an underscore: the HTML reporter shows every annotation except those whose type starts with _.

A quarantine reporter

A known flake should pass, and a new error in the same test should not. A reporter sees every attempt of every test once the run is over, so the decision goes there.

Anti-pattern

Clear every quarantined failure

The quick version waits for a failed run, checks that every failing test carries the tag, and flips the status to passed:

// Anti-pattern: any failure of a quarantined test turns the run green, a real regression included
import type { FullConfig, FullResult, Reporter, Suite } from '@playwright/test/reporter';

export default class LenientQuarantineReporter implements Reporter {
  private suite?: Suite;

  onBegin(_config: FullConfig, suite: Suite) {
    this.suite = suite;
  }

  async onEnd(result: FullResult) {
    const failing = (this.suite?.allTests() ?? []).filter(
      (test) => test.outcome() === 'unexpected',
    );
    if (result.status === 'failed' && failing.every((test) => test.tags.includes('@quarantine'))) {
      return { status: 'passed' as const };
    }
    return undefined;
  }
}

That reporter can't tell a flake from a break. If the checkout API starts returning errors, the quarantined checkout test fails every attempt, the reporter clears it, and CI goes green on a real regression.

Fix

Never let the reporter turn a failed run green. The version below only adds failures: it leaves known flakes alone and turns the run red for everything else.

The reporter

A custom reporter is a class that implements some of the reporter methods. This one keeps the root suite from onBegin, and the reporter judges each test once in onEnd, after every attempt has run. Three APIs do the sorting. testCase.outcome() gives the test's outcome, which is not the same as testResult.status, and testCase.results holds a result for each run of the test. testResult.errors holds the errors thrown during the test execution. The rules:

  • A quarantined test that comes out flaky, with every error matching its signature, passes.
  • A quarantined test that fails every attempt fails the run.
  • A quarantined test that throws an error its signature doesn't match, in any attempt, fails the run.
  • A test outside quarantine that comes out flaky fails the run.
  • A broken quarantine record, or more quarantined tests than the cap, fails the run.
import type {
  FullConfig,
  FullResult,
  Reporter,
  Suite,
  TestCase,
} from '@playwright/test/reporter';
import { capProblem, recordProblems } from './quarantine-record';

type Options = { maxQuarantined?: number };

const isQuarantined = (test: TestCase) => test.tags.includes('@quarantine');

export default class QuarantineReporter implements Reporter {
  private suite?: Suite;

  constructor(private options: Options = {}) {}

  onBegin(_config: FullConfig, suite: Suite) {
    this.suite = suite;
  }

  async onEnd(_result: FullResult) {
    const tests = this.suite?.allTests() ?? [];
    const problems: string[] = [];

    for (const test of tests) {
      const name = test.titlePath().slice(1).join(' › ');
      const outcome = test.outcome();

      if (!isQuarantined(test)) {
        // Outside quarantine, a flake is a new problem
        if (outcome === 'flaky') problems.push(`${name}: flaky outside quarantine`);
        continue;
      }

      problems.push(...recordProblems(test).map((p) => `${name}: ${p}`));

      if (outcome === 'unexpected') {
        problems.push(`${name}: failed every attempt`);
        continue;
      }

      const signature = test.annotations.find(
        (a) => a.type === 'quarantine-signature',
      )?.description;
      const errors = test.results.flatMap((r) => r.errors);
      const unmatched = errors.find(
        (e) => !signature || !e.message?.includes(signature),
      );
      if (unmatched) {
        problems.push(`${name}: unmatched error: ${unmatched.message?.split('\n')[0]}`);
      }
    }

    // Count each quarantined test once, across projects and repeats
    const quarantinedIds = new Set(
      tests.filter(isQuarantined).map((t) => t.titlePath().slice(2).join(' › ')),
    );
    const cap = capProblem(quarantinedIds.size, this.options.maxQuarantined);
    if (cap) problems.push(cap);

    if (problems.length === 0) return undefined;
    for (const problem of problems) console.error(`[quarantine] ${problem}`);
    // Only ever adds failures; never turns a failed run green
    return { status: 'failed' as const };
  }
}

A reporter is allowed to override the run status, and that changes the runner's exit code. When someone asked for a way to let reporters change the exit code, a contributor replied that the reporter's end-of-run hook can already override it (#40414). The request was closed as not planned. onEnd is where a reporter may override the status and so affect the exit code of the test runner. Failures outside quarantine fail the run without it; the reporter adds the quarantined failures that look real.

Expiry, owners and caps

An annotation nobody checks goes as stale as a bare tag. The reporter calls two checks from quarantine-record.ts, one per quarantined test and one for the whole set:

import type { TestCase } from '@playwright/test/reporter';

const REQUIRED = [
  'quarantine-owner',
  'quarantine-issue',
  'quarantine-expires',
  'quarantine-signature',
];

export function recordProblems(test: TestCase, now = new Date()): string[] {
  const get = (type: string) =>
    test.annotations.find((a) => a.type === type)?.description;
  const problems = REQUIRED.filter((type) => !get(type)).map((type) => `missing ${type}`);

  const issue = get('quarantine-issue');
  if (issue && !issue.startsWith('https://')) {
    problems.push(`quarantine-issue is not a link: ${issue}`);
  }

  const expires = get('quarantine-expires');
  // An unparseable date compares false, so it counts as expired too
  if (expires && !(new Date(expires) > now)) {
    problems.push(`expired on ${expires}: fix it, renew it or delete it`);
  }
  return problems;
}

export function capProblem(count: number, max?: number): string | undefined {
  if (max === undefined || count <= max) return undefined;
  return `${count} tests in quarantine, cap is ${max}: fix one before adding another`;
}

recordProblems fails a test with a missing annotation, an issue that isn't a link, or an expiry date that has passed or doesn't parse. capProblem fails the run when more tests sit in quarantine than the reporter's maxQuarantined option allows.

We'd start the cap at 10, the value the config in the next section passes. Every quarantined test is one whose known failure no longer fails the build, and we keep that list short enough that one reviewer can read every record, owner and expiry date in a single sitting. Past the cap, adding the next test means fixing one first, so quarantine can't turn into the place tests go to die. Lower the cap as the list shrinks, and raise it only with a plan to bring it back down. A cap that moves every time someone hits it has stopped being a cap.

Treat an expired entry as an escalation. The owner fixes the test, renews the quarantine with a new date and a reason in the issue, or deletes the test. Quietly bumping the date is the one move to refuse in review.

When the CI environment itself is broken, don't quarantine the whole suite. That's an outage: fix the environment, and keep quarantine for tests that flake on a healthy one.

failOnFlakyTests and quarantined tests

Retries hide flakiness. The --fail-on-flaky-tests option, which fails the run if any test is flagged as flaky, is off by default. failOnFlakyTests makes the run exit with an error if any test is marked flaky, and the docs call it useful on CI. That includes quarantined tests: when any of them is marked flaky in a run, the run exits with an error.

Don't wait for a softer version. A request for a setting that fails the run once more than a set number of tests are flaky was closed for limited engagement (#37002).

Anti-pattern

FailOnFlakyTests over quarantined tests

The blocking job turns on failOnFlakyTests while quarantined tests still run in it. Every known flake fails every build it flakes in, the quarantine stops protecting anyone, and the setting gets switched off for the whole suite, new flakes included.

Fix

In a single run, leave failOnFlakyTests off and let the quarantine reporter fail on flaky tests outside quarantine. Turn it on only in a job that runs nothing but quarantined tests, like the nightly check below. Register the quarantine reporter next to the HTML reporter. The exit code and the report then come from the same run:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  repeatEach: Number(process.env.REPEAT_EACH ?? 1),
  // On only in the nightly job, which runs nothing but quarantined tests
  failOnFlakyTests: process.env.QUARANTINE_NIGHTLY === '1',
  reporter: [
    ['html'],
    ['./quarantine-reporter.ts', { maxQuarantined: 10 }],
    ['./quarantine-list-reporter.ts'],
  ],
});

Leave maxQuarantined out of sharded jobs, for the reason in the next section.

Sharded runs and merged reports

Sharding splits the suite, and the cap needs the whole suite in view. The per-test rules don't: the reporter runs inside each shard, so each shard's exit code applies those rules to its own tests, and a flaky test outside quarantine still fails the shard it ran in.

The cap is different. A shard sees only its own part of the suite, so a cap checked per shard counts only the quarantined tests that shard ran, and every shard can pass while the total sits well above the cap. Run the count once, over the merged result instead: each test shard has its own test report, and a combined report showing all the results needs the shards' reports merged (covered in how a reporter sees a sharded run). Blob reports from sharded runs can be merged and converted into any other Playwright report, including the JSON report, which holds all the information about the run in one object. Count the distinct tests tagged @quarantine in that JSON and fail the merge step when the count is above the cap.

Budget shard time for quarantine too. Tests statically skipped with test.fixme() aren't counted when balancing shards, because they don't run. Quarantined tests do run, so they take shard time like any other test, plus the retries they use when they flake.

Nightly exit checks

Quarantine needs a way out, and "it's been green for a while" isn't one. A nightly job that runs only the quarantined tests, many times each, gives every test a failure rate instead of a guess.

A second small reporter writes the list. --test-list reads a file that lists tests in a format similar to the --list output, so the reporter writes one line per quarantined test:

import { writeFileSync } from 'node:fs';
import type { FullConfig, Reporter, Suite } from '@playwright/test/reporter';

export default class QuarantineListReporter implements Reporter {
  onBegin(_config: FullConfig, suite: Suite) {
    const lines = suite
      .allTests()
      .filter((test) => test.tags.includes('@quarantine'))
      .map((test) => {
        const [, project, file, ...titles] = test.titlePath();
        const prefix = project ? `[${project}] › ` : '';
        return `${prefix}${file} › ${titles.join(' › ')}`;
      });
    // A Set drops the duplicates that repeats would add
    writeFileSync('quarantine-list.txt', [...new Set(lines)].join('\n'));
  }
}

Compare a few lines against npx playwright test --list the first time you run it. Keep quarantine-list.txt from the main branch's latest run as a CI artifact, and have the nightly job run that list with repeatEach and with failOnFlakyTests on. A shell check skips the run when the list is empty:

if [ -s quarantine-list.txt ]; then
  npx playwright test --test-list quarantine-list.txt --repeat-each=20 --fail-on-flaky-tests
else
  echo "No quarantined tests, skipping the nightly check"
fi

When retries are on, a repetition that fails first and passes on a retry is flaky, and --fail-on-flaky-tests makes the run fail when any test is flaky. A green exit is a clean night for every test on the list. On a red one, the HTML report shows which test broke its streak.

Set the exit rule before the first night. We'd release a test after seven consecutive clean nights with --repeat-each at 20, which is 140 repetitions in a row without a flaky or failed result. One clean night proves little, because a test that still flakes now and then can get through a single night's repetitions without a miss. A week also spans a full cycle of weekday and weekend runs, so a flake tied to one day's load or deploys gets a chance to show. A red night resets the streak to zero, however close the test was.

When the streak is complete, the owner removes the test's tag and annotations in one commit. A test that hasn't earned that by its expiry date gets fixed or deleted. We'd set every new expiry date at least two weeks out, so a fix has time to land and still earn its week before the date arrives.

New end-to-end tests can go through the same check. Add them quarantined, let the nightly job run them, and make them gate merges only once they pass it.

Checking it works

Before the reporter judges real tests, give it four it should get right. Put them in their own spec file, run it once with retries on, and delete it:

import { test } from '@playwright/test';

// Owner and issue are valid; only the signature and expiry vary
const record = (signature: string, expires = '2099-01-01') => [
  { type: 'quarantine-owner', description: 'qa-platform' },
  { type: 'quarantine-issue', description: 'https://github.com/acme/shop/issues/500' },
  { type: 'quarantine-expires', description: expires },
  { type: 'quarantine-signature', description: signature },
];

test('simulated flake', { tag: '@quarantine', annotation: record('simulated flake') }, async ({}, testInfo) => {
  // Throws on the first attempt only, so the retry passes
  if (testInfo.retry === 0) throw new Error('simulated flake');
});

test('unmatched error', { tag: '@quarantine', annotation: record('simulated flake') }, async ({}, testInfo) => {
  if (testInfo.retry === 0) throw new Error('payment API returned 500');
});

test('broken for real', { tag: '@quarantine', annotation: record('simulated flake') }, async () => {
  throw new Error('simulated flake');
});

test('expired entry', { tag: '@quarantine', annotation: record('simulated flake', '2020-01-01') }, async () => {});
CI=1 npx playwright test tests/quarantine-check.spec.ts
  • simulated flake throws only on its first attempt, when testInfo.retry is zero, so it passes when retried and Playwright counts it as flaky. Its one error matches its signature, and the reporter prints nothing for it.
  • unmatched error also passes on its retry, but its first attempt threw payment API returned 500, which the signature doesn't match. The reporter prints an unmatched-error line.
  • broken for real fails on the first run and on every retry, so Playwright counts it as failed. The reporter prints a failed-every-attempt line.
  • expired entry passes, but its expiry date is in the past, and the reporter prints an expiry line.

The run fails. Delete the last three tests and run it again: with only the simulated flake left, the build passes.

Gotchas and limits

Most ways this setup goes wrong come from making quarantined tests easier to pass than the rest of the suite.

Anti-pattern

Extra retries for quarantined tests

Retries can be set for one group of tests or a single file with test.describe.configure(), and a file of quarantined tests looks like a good place to raise them:

// Anti-pattern: five retries make "failed every attempt" rare, so a real break comes out flaky
import { test, expect } from '@playwright/test';

test.describe.configure({ retries: 5 });

test(
  'submits an order',
  {
    tag: '@quarantine',
    annotation: [
      { type: 'quarantine-owner', description: 'payments-team' },
      { type: 'quarantine-issue', description: 'https://github.com/acme/shop/issues/482' },
      { type: 'quarantine-expires', description: '2026-12-15' },
      { type: 'quarantine-signature', description: 'Order confirmed' },
    ],
  },
  async ({ page }) => {
    await page.goto('https://example.com/checkout');
    await page.getByRole('button', { name: 'Place order' }).click();
    await expect(page.getByText('Order confirmed')).toBeVisible();
  },
);

More attempts mean more chances to pass. A test that has started failing far more often than before still tends to pass one of them, so it comes out flaky instead of failed, and the failed-every-attempt rule rarely fires.

Fix

Delete the test.describe.configure() line. Quarantined tests get the same retries as the rest of the suite, so failing every attempt means the same thing for both.

Leaked state behind retries

When a test fails, Playwright discards the worker process along with the browser and starts a new one. With retries enabled, the second worker process starts by retrying the failed test and continues from there. When a test flakes only in the full suite, repeat the alone-versus-with-its-file check before anyone renews it.

Runtime annotations

Annotations can also be added while a test runs, through test.info().annotations. Keep quarantine records static in the declaration anyway: a reviewer audits a static annotation by reading the test, and a runtime one only by running it.

Where hand-rolled quarantine stops

Everything above lives in the repository, which is both its strength and its limit. Every quarantine change is a commit. A stale branch carries old annotations, a renewal waits for review, and a test's quarantine history lives in git rather than in a trend line.

Past a certain suite size, quarantine and release should follow each test's run history rather than a code change. The quarantine action in Currents Actions runs the test and ignores its failures in the client, after the test executes and before its results are reported; the results still go to Currents, and errors in Playwright's beforeAll hook are not covered, so those tests are still reported as failed. For the way out, Test Explorer reports each test's Flakiness Rate, the percentage of executions with a non-deterministic outcome, and its % Ignored, the percentage of executions where the test was skipped, ignored or marked as muted. In its % Flaky Trend, a negative value means a test's stability has improved.

Whichever version you run, keep one rule: a test that fails every time was never a quarantine candidate. Fix it or delete it.

Frequently asked questions

How do I quarantine flaky Playwright tests?

Keep the tests in the normal run with a tag, and decide what their results mean in a reporter after the run ends. Quarantine works when the test stays in the normal run, has an owner, an issue and an expiry date, and still fails the build when its failure looks real. The fixme and grep-invert anti-pattern shows what goes wrong when you take the test out of the run instead.

Why not use test.fixme to quarantine a flaky test?

test.fixme() marks a test as failing, and Playwright doesn't run it. You can't watch the test recover, and a regression in the code it covers ships green. --grep-invert has the same problem, as the fixme and grep-invert anti-pattern explains.

Should I use test.fail for flaky tests?

No. test.fail() fits a test you expect to fail every time. Playwright runs the test, checks that it fails and complains if it passes, so a flaky test that passes most of the time would make Playwright complain on most runs, as the options side by side shows.

How do I confirm a test is really flaky before quarantining it?

Measure before you tag: run the suspect test alone with repeatEach, then with the rest of its file. Failures when the test runs alone mean it is intermittent on its own. A test that is clean alone but red with its file points to order dependence or leaked state, which needs a fix in the tests, not a quarantine, as quarantine after one red run explains.

Can a Playwright reporter change the exit code?

Yes. A reporter is allowed to override the run status, and that changes the runner's exit code.

Should failOnFlakyTests be on when quarantined tests run?

No. In a single run, leave failOnFlakyTests off and let the quarantine reporter fail on flaky tests outside quarantine. Turn it on only in a job that runs nothing but quarantined tests, like the nightly check, as the failOnFlakyTests anti-pattern explains.

How does quarantine work with sharded Playwright runs?

The reporter runs inside each shard, so each shard's exit code applies the per-test rules to its own tests.

When should a flaky test leave quarantine?

A test should leave quarantine on evidence from a nightly job that runs only the quarantined tests many times each, which gives every test a failure rate instead of a guess. With retries on, a repetition that fails first and passes on a retry is flaky, and --fail-on-flaky-tests makes the run fail when any test is flaky, so a green exit is a clean night for every test on the list, as the nightly exit checks show.