Playwright Test Observability: Run History, Flakes, Traces

A test fails on Tuesday, passes on the retry, and the run goes green. On Friday it fails on another branch with what looks like the same timeout. Someone asks whether it has been flaky for a month or only since last week's merge, and nobody can say.

One run's results tell you what failed in that run. They can't tell you whether a test is getting slower, whether it fails more on one branch than another, or which commit it started on. The evidence expires, too: on GitHub Actions, build logs and artifacts are stored for 90 days by default, and this retention period can be customized.

A production service with this problem would get per-request events, metrics and traces, connected over time. Give the suite the same treatment and connect them across runs. A pass-rate threshold on one run tells you the run was bad. History tells you why, and whether it's getting worse.

The fix is a small pipeline you own. Every test attempt becomes a stored event, and a failed attempt links to its own backend trace.

One event per test attempt

The unit you store decides which questions you can answer later. Pick the wrong one and no query recovers what you threw away.

Playwright gives each test an ID and keeps a record of each attempt. testCase.id is computed from the test file name, the test title and the project name, and it is unique within a Playwright session. We use the ID as the join key for a test's history.

testCase.results holds the results for each run of the test, and testResult.retry gives each retry attempt a sequential number. The attempt is the natural unit for an event.

testCase.outcome() is where the detail disappears. It reports a test that passes on a second retry as 'flaky', a single label that carries neither the failed attempt's error nor its duration.

Failing tests aren't retried by default, and Playwright categorizes a test as flaky when it failed on the first run but passed when retried. Without retries, a flaky test never gets the second attempt that would earn it that label, so it looks like a plain failure. Turn retries on in CI if you want flake detection inside a run.

When a test fails, Playwright Test discards the whole worker process along with its browser and starts a new one. A retry therefore starts clean, in a fresh worker. Record it as its own event, the way a tracing system records each request as its own span.

A structured event reporter

A Playwright contributor recommends a custom reporter for this (#27147). The thread opened with a request for OpenTelemetry tracing in Playwright Test to find which tests were slow. The contributor recommended a custom reporter to get the test and step data, and said having it inside Playwright was outside the project's scope. The request was closed as not planned.

A custom reporter is a class that implements some of the reporter methods Playwright calls during a run. This reporter uses four:

• onBegin is called once before tests run, when all tests have been discovered and arranged into suites. Capture run-level context there.
• onTestEnd is called after a test finishes in the worker process. Build the attempt's event there.
• onEnd is called after all tests have run or testing was interrupted, and Playwright Test awaits the Promise it returns. Flush there.
• printsToStdio returns false, as the docs strongly recommend for a reporter that doesn't print to the terminal.

If you want a live view, you can send events from onTestEnd as tests finish, as long as onEnd waits for every request you started. Otherwise, buffer them and send once.

// Anti-pattern: no code waits for these requests, so nothing makes sure they arrive
import type { Reporter, TestCase, TestResult } from '@playwright/test/reporter';

class NaiveReporter implements Reporter {
  onTestEnd(test: TestCase, result: TestResult) {
    fetch(process.env.TEST_EVENTS_URL!, {
      method: 'POST',
      body: JSON.stringify({ testId: test.id, status: result.status }),
    });
  }

  onEnd() {}
}

export default NaiveReporter;

// reporters/events-reporter.ts
import type {
  FullConfig,
  Reporter,
  TestCase,
  TestResult,
} from '@playwright/test/reporter';
import { errorSignature } from './error-signature';

type AttemptEvent = {
  runId: string;
  testId: string;
  titlePath: string[];
  retry: number;
  status: string;
  startedAt: string;
  durationMs: number;
  errorMessage: string | null;
  errorSignature: string | null;
  parallelIndex: number;
  shard: number | null;
  commitSha: string;
  branch: string;
  runner: string;
  backendTraceUrl: string | null;
};

class EventsReporter implements Reporter {
  private events: AttemptEvent[] = [];
  private metadata: Record<string, unknown> = {};
  private shard: number | null = null;

  onBegin(config: FullConfig) {
    // Run-level context, the same for every attempt this machine reports
    this.metadata = config.metadata;
    this.shard = config.shard?.current ?? null;
  }

  onTestEnd(test: TestCase, result: TestResult) {
    const backendTrace = result.attachments.find((a) => a.name === 'backend-trace');

    this.events.push({
      runId: String(this.metadata.ciRunId ?? 'local'),
      testId: test.id,
      titlePath: test.titlePath(),
      retry: result.retry,
      status: result.status,
      startedAt: result.startTime.toISOString(),
      durationMs: result.duration,
      errorMessage: result.errors[0]?.message ?? null,
      errorSignature: errorSignature(result.errors),
      parallelIndex: result.parallelIndex,
      shard: this.shard,
      commitSha: String(this.metadata.commitSha ?? ''),
      branch: String(this.metadata.branch ?? ''),
      runner: String(this.metadata.runner ?? ''),
      backendTraceUrl: backendTrace?.body?.toString('utf8') ?? null,
    });
  }

  async onEnd() {
    const url = process.env.TEST_EVENTS_URL;
    if (!url || this.events.length === 0) return;

    // Playwright awaits the Promise that onEnd returns
    const response = await fetch(url, {
      method: 'POST',
      headers: {
        'content-type': 'application/json',
        authorization: `Bearer ${process.env.TEST_EVENTS_TOKEN}`,
      },
      body: JSON.stringify(this.events),
    });

    if (!response.ok) {
      process.stderr.write(`events upload failed: ${response.status}\n`);
    }
  }

  printsToStdio() {
    return false;
  }
}

export default EventsReporter;

Running it across shards

The reporter only pays off if every shard runs it and every event says where it ran.

# Anti-pattern: the reporter exists only in this one CI command
npx playwright test --reporter=./reporters/events-reporter.ts

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  reporter: [
    ['list'],
    ['html'],
    ['./reporters/events-reporter.ts'],
  ],
  metadata: {
    ciRunId: process.env.CI_RUN_ID ?? 'local',
    commitSha: process.env.CI_COMMIT_SHA ?? '',
    branch: process.env.CI_BRANCH ?? '',
    runner: process.env.CI_RUNNER ?? '',
  },
});

One run, many machines

The reporter reads the shard index from config.shard.current and the run ID from metadata. In GitHub Actions, the matrix passes the same values and the store's credentials to every shard:

# .github/workflows/e2e.yml
name: e2e
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shardIndex: [1, 2, 3, 4]
        shardTotal: [4]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: lts/*
      - run: npm ci
      - run: npx playwright install --with-deps
      - name: Run shard
        run: npx playwright test --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}
        env:
          CI_RUN_ID: ${{ github.run_id }}-${{ github.run_attempt }}
          CI_COMMIT_SHA: ${{ github.sha }}
          CI_BRANCH: ${{ github.head_ref || github.ref_name }}
          CI_RUNNER: ${{ runner.name }}
          TEST_EVENTS_URL: ${{ secrets.TEST_EVENTS_URL }}
          TEST_EVENTS_TOKEN: ${{ secrets.TEST_EVENTS_TOKEN }}

On GitLab or any other CI, map the equivalent values into the same variables and the reporter doesn't change. Keep the store's write credentials in CI secrets, as the workflow does.

Shard and runner signals

Per-test durations grouped by shard show which shard is the long pole in each run. That table is the input duration-based shard balancing needs, whether you tune the split by hand or hand it to a tool. The same rows list the tests that failed in a run, and a job that reruns only the failures needs nothing more.

Grouping failures by runner and shard separates a broken machine from a broken test. When every test on one runner fails with the same error signature while the other shards pass, check the machine before you touch the tests.

Storage that outlasts CI artifacts

History has to outlive a CI artifact's retention period: GitHub Actions, for example, stores artifacts for 90 days by default.

-- One row per attempt; the primary key makes re-uploads harmless
CREATE TABLE test_attempts (
  run_id               TEXT        NOT NULL,
  test_id              TEXT        NOT NULL,
  retry                INTEGER     NOT NULL,
  title_path           TEXT[]      NOT NULL,
  status               TEXT        NOT NULL,
  started_at           TIMESTAMPTZ NOT NULL,
  duration_ms          INTEGER     NOT NULL,
  error_message        TEXT,
  error_signature      TEXT,
  parallel_index       INTEGER     NOT NULL,
  shard                INTEGER,
  commit_sha           TEXT        NOT NULL,
  branch               TEXT        NOT NULL,
  runner               TEXT        NOT NULL,
  backend_trace_url    TEXT,
  playwright_trace_url TEXT,
  PRIMARY KEY (run_id, test_id, retry)
);

INSERT INTO test_attempts (
  run_id, test_id, retry, title_path, status, started_at, duration_ms,
  error_message, error_signature, parallel_index, shard,
  commit_sha, branch, runner, backend_trace_url
)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15)
ON CONFLICT (run_id, test_id, retry) DO NOTHING;

What breaks the history

Because testCase.id is computed from the file name, the test title and the project name, renaming a test or moving it to another file starts a new history under a new ID. Store titlePath(), the list of titles from the root down to the test, next to the ID. When one history stops and another starts with a near-identical title path in the same run, merge them.

The project name is part of the ID as well, so each browser or device project builds its own history. A test that is stable in one browser project and flaky in another shows up as two IDs with two flake rates, and you see the cross-browser difference without extra work.

Compare the IDs in consecutive runs on the same branch, too. A test that silently stopped running never fails, so no pass rate will flag it.

Dashboards and audit trail

Point the dashboards your team already uses, Grafana or Datadog, at the table instead of building a new UI, and schedule a weekly test-health query that posts where people will see it. The append-only table doubles as an audit trail: every attempt, on every branch, with the commit it ran against.

Flakiness, p95 and failure signatures

With the attempts stored, four numbers answer most of the cross-run questions a single report leaves open.

Two kinds of flake

Within a run, a test is flaky when it failed on the first run and passed when retried, the definition Playwright itself uses. Across runs, a test is flaky when it passed in one run and failed in another on the same commit, whether or not it retried. Compute both over a rolling window, so one bad week fades out instead of branding a test for good.

-- Tests that both passed and failed on the same commit in the last 30 days
SELECT test_id, commit_sha, count(*) AS runs
FROM (
  SELECT test_id, run_id, commit_sha, bool_or(status = 'passed') AS passed
  FROM test_attempts
  WHERE started_at >= now() - interval '30 days' AND status IN ('passed', 'failed', 'timedOut')
  GROUP BY test_id, run_id, commit_sha
) per_run
GROUP BY test_id, commit_sha
HAVING bool_or(passed) AND NOT bool_and(passed);

Retry rate is the share of a test's runs that needed at least one retry. Each event records testResult.retry, so the store can compute it. A one-row-per-test schema can't.

-- Per test: last 30 days vs the 30 before
WITH per_run AS (
  SELECT
    test_id,
    run_id,
    min(started_at) >= now() - interval '30 days' AS is_current,
    bool_or(status = 'passed') AND bool_or(status IN ('failed', 'timedOut')) AS flaky,
    max(retry) > 0 AS retried
  FROM test_attempts
  WHERE started_at >= now() - interval '60 days'
  GROUP BY test_id, run_id
),
run_stats AS (
  SELECT
    test_id,
    is_current,
    count(*) AS runs,
    sum(flaky::int) AS flaky_runs,
    avg(flaky::int) AS flake_rate,
    avg(retried::int) AS retry_rate
  FROM per_run
  GROUP BY test_id, is_current
),
p95 AS (
  -- Passing attempts only: timed-out failures would inflate the tail
  SELECT
    test_id,
    started_at >= now() - interval '30 days' AS is_current,
    percentile_cont(0.95) WITHIN GROUP (ORDER BY duration_ms) AS p95_ms
  FROM test_attempts
  WHERE status = 'passed'
    AND started_at >= now() - interval '60 days'
  GROUP BY test_id, is_current
)
SELECT
  cur.test_id,
  cur.runs,
  cur.flaky_runs,
  round(cur.flake_rate, 3) AS flake_rate,
  round(prev.flake_rate, 3) AS prev_flake_rate,
  round(cur.retry_rate, 3) AS retry_rate,
  p_cur.p95_ms,
  p_prev.p95_ms AS prev_p95_ms
FROM run_stats cur
LEFT JOIN run_stats prev ON prev.test_id = cur.test_id AND NOT prev.is_current
LEFT JOIN p95 p_cur ON p_cur.test_id = cur.test_id AND p_cur.is_current
LEFT JOIN p95 p_prev ON p_prev.test_id = cur.test_id AND NOT p_prev.is_current
WHERE cur.is_current
ORDER BY cur.flaky_runs DESC;

Failure signatures

Raw error messages don't group: two failures from the same broken endpoint can differ by a request ID, a port and a timing. testResult.errors holds the errors thrown during the test's execution. Normalize the first one into a signature before you store it.

// reporters/error-signature.ts
import type { TestError } from '@playwright/test/reporter';

export function errorSignature(errors: TestError[]): string | null {
  const message = errors[0]?.message;
  if (!message) return null;

  return message
    .replace(/\u001b\[[0-9;]*m/g, '') // drop terminal color codes
    .split('\n')[0]
    .replace(/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/gi, '<uuid>')
    .replace(/\b[0-9a-f]{16,}\b/gi, '<hex>')
    .replace(/\d+(\.\d+)?/g, '<n>')
    .trim()
    .slice(0, 300);
}

Group a run's failures by signature and a wall of red can collapse into one backend error. You get failure classification you can read and diff.

From numbers to decisions

Rank fixes by how many runs a test disrupted, not by flake rate alone. A test that flakes rarely but runs on every PR can break more builds than one that flakes often in a nightly job. Use a flake-rate threshold to send a test to quarantine, and fail the PR gate when a branch introduces an error signature that main has never produced.

Keeping traces of failed attempts

Metrics say which test to open. The trace says what happened.

With trace set to 'retain-on-failure', Playwright records a trace for every run but keeps it only for runs that failed, and a failed run's trace is kept even when a later retry passes. A flaky test needs exactly that trace. Pair 'retain-on-failure' with retries that are on in CI and off locally:

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  // On in CI so flaky tests get a second attempt; off locally so failures stay fast
  retries: process.env.CI ? 2 : 0,
  use: {
    trace: 'retain-on-failure',
  },
});

Upload retained traces to object storage and write each URL into its attempt's playwright_trace_url column, so the trace lives as long as the event does. The Trace Viewer can open a remote trace by URL, so nobody has to download the file from the CI run first. A flaky test's failed row then links straight to its trace, ready to compare against the passing retry.

Linking failures to backend traces

The Playwright trace shows the network requests made during your test, with their headers and bodies (Trace Viewer). When the failure is a 500 from a service behind the API, you need the backend's trace for that exact request.

The link is a trace ID sent as a header. extraHTTPHeaders sends additional HTTP headers with every request and defaults to none, so a trace ID set there reaches the app on every request the test makes. Make sure your backend's tracing reads that header, so its spans and logs carry the test's ID.

Renew the ID on every attempt. One team creates a Datadog APM trace ID for each test run (#31284). They renew it on every retry, because a retry is a new session. They asked for links in the HTML report's attachments. A maintainer folded the request into another issue, and this one was closed as completed, labelled v1.46.

// Anti-pattern: every test and every retry in the run sends the same trace ID
// playwright.config.ts
import { defineConfig } from '@playwright/test';

const runTraceId = process.env.CI_TRACE_ID ?? '4bf92f3577b34da6a3ce929d0e0e4736';

export default defineConfig({
  use: {
    extraHTTPHeaders: {
      traceparent: `00-${runTraceId}-00f067aa0ba902b7-01`,
    },
  },
});

// fixtures/trace-context.ts
import { test as base } from '@playwright/test';
import { randomBytes } from 'node:crypto';

export const test = base.extend({
  extraHTTPHeaders: async ({ extraHTTPHeaders }, use, testInfo) => {
    // A new ID every time this runs, so each attempt gets its own backend trace
    const traceId = randomBytes(16).toString('hex');
    const spanId = randomBytes(8).toString('hex');

    await testInfo.attach('backend-trace', {
      body: `https://tracing.example.com/trace/${traceId}`,
      contentType: 'text/plain',
    });

    await use({ ...extraHTTPHeaders, traceparent: `00-${traceId}-${spanId}-01` });
  },
});

export { expect } from '@playwright/test';

Third-party requests

extraHTTPHeaders goes out with every request, so third-party origins the page calls receive the trace header too. If that matters, strip the header at the edge of your test environment.

From red test to root cause

A failed event now leads to two places: the Playwright trace for the browser side and the backend trace for the request that broke. The backend side opens in the tracing tool your team already uses for production traffic, so a test failure reads like any other incident.

When DIY stops paying off

The reporter is the cheap part. The schema, the error-normalization rules that drift as messages change, the dashboards and the rename stitching are what you maintain for as long as the suite lives.

Choose by what you already run and who will own it. For a small suite with one owner, built-in reporters with retries are enough: Playwright already labels in-run flakes, and building this pipeline isn't worth it. Build the DIY pipeline only if you already run a database and a dashboard stack, and someone owns the schema and the signature rules; otherwise the pipeline turns into a second product to maintain. A hosted tool fits when you want cross-run history, flake rates and error grouping without owning the upkeep.

If you'd rather not build the pipeline, Currents Test Explorer computes each test's flakiness rate as flaky results divided by selected results, and its Duration Trend compares a test's average duration across two equal periods, such as the last 30 days against the previous 30. Currents Error Explorer parses each failed test's raw error message and stack trace into structured fields, and its Action, Target and Category fields reveal failure patterns across CI runs, the grouping your signature rules would otherwise do.

Where to start

Start with the per-attempt reporter, the store and the flake-rate query. A week of data is enough for those three to answer most cross-run questions. Add retained Playwright traces next, then per-attempt backend trace IDs once your app's tracing reads the header.

Every week, review the flakiest tests, the biggest p95 growth and the newest error signatures, and let the data decide what goes to quarantine. The next time someone asks whether that Tuesday timeout has been flaky for a month, the answer is a query, not a guess. Memory says a test has always been a bit flaky. The table says since which commit.

Frequently asked questions