Agentic E2E Testing with Playwright: What Holds Up in CI

Coding agents made feature work faster, and the E2E suite didn't keep pace. Every new screen still needs a spec, and every UI change still breaks a few old ones. So the test work is moving to agents too: one explores the app, one writes the test, one repairs it when CI goes red.

That is where the trouble starts. Playwright MCP operates on the page's accessibility tree, not pixels: each tool it runs returns a structured snapshot of the page's elements, their roles and text content. A merge gate has to give the same verdict for the same commit every time, or a red build stops meaning anything.

What makes testing agentic is who decides the next step. In AI-assisted testing, a human decides and a model suggests code. In AI-generated testing, a model writes the whole test in one shot. In agentic testing, an agent loops: it reads the app's state, acts, and decides again.

Playwright comes with three Test Agents out of the box: a planner, a generator and a healer. The generator transforms the Markdown plan into Playwright Test files, and the healer automatically repairs failing tests. The Playwright docs say the agents can run on their own, one after another, or chained in an agentic loop, and that running them in sequence produces test coverage for your product. Whether to use them is the easy question. Where each one runs in CI is the hard one.

Get the placement right, and agents author and repair tests off the merge path while only reviewed, committed specs decide whether a PR merges.

Where agents fit in CI

Every agent belongs somewhere other than the merge gate. Each kind gets a place in the pipeline where a person reviews its output before that output can touch a merge:

Agents propose. A plain Playwright run with no model in it decides.

Live checks are not verdicts

The generator checks its work as it goes. Playwright's docs say it turns the Markdown plan into executable Playwright tests and verifies selectors and assertions live while it performs the scenarios. The same docs say generated tests may still include initial errors, which the healer can fix automatically. So the live check is part of writing the test. It is not the verdict.

Goals and fixed paths

"A user can upgrade to Pro" is a goal, and an agent can reach it by whatever route the page offers. "Open billing, pick the annual plan, confirm, see the Pro badge" is a fixed path, and a spec that walks it fails the moment one step changes. Keep critical paths as deterministic specs. Agents belong in the layer above them, exploring what the specs don't cover and debugging what they catch.

Autonomy per step

Core regression that runs on every commit stays fully specified, with every locator and assertion written down. Give an agent a bounded step only where user state or a feature flag changes the path, and pin the outcome with an exact assertion right after it.

Adoption order

Start with repair and triage, move to generation on new surfaces, and add exploration last. A repair starts from a real CI failure with a trace behind it. Generation starts from a plan someone reviewed. Exploration starts from a request and a seed test, so it has the least evidence behind it and earns trust last.

How agents see the page

An agent never sees the page the way your users do. Playwright MCP works on the page's accessibility tree, not pixels: each tool returns a structured snapshot of the page's elements with their roles and text content. The agent's picture of the page is that tree, as of the last snapshot.

Elements in a snapshot carry refs such as [ref=e2]. Refs stay stable within one snapshot, but the page changing invalidates them, so an agent should re-snapshot after every navigation. An agent that skips that step acts on refs from a page that no longer exists.

Refs in that form also show up inside a test. page.ariaSnapshot() reports refs like [ref=e2] when called in "ai" mode, and those refs resolve against the latest snapshot taken in the element's frame. When an agent-written test goes wrong at a step, capture that snapshot at the step and compare it with what the agent acted on:

import { test, expect } from '@playwright/test';

test('applies a discount code at checkout', async ({ page }, testInfo) => {
  // assumes use: { baseURL } is set in playwright.config.ts
  await page.goto('/cart');
  await page.getByRole('button', { name: 'Apply code' }).click();

  // The agent-written test diverged here: record what an agent would see
  const snapshot = await page.ariaSnapshot({ mode: 'ai' });
  await testInfo.attach('aria-snapshot-ai', { body: snapshot, contentType: 'text/plain' });

  await page.getByRole('textbox', { name: 'Discount code' }).fill('SPRING10');
  await expect(page.getByTestId('cart-total')).toHaveText('$45.00');
});

Read the attachment next to the agent's transcript. If the agent targeted a role or name that isn't in the snapshot at that step, it worked from an older page.

MCP or CLI

Use MCP for interactive authoring, where you watch the agent and steer it. Use the Playwright CLI where tokens matter, because its concise output avoids loading large tool schemas into the model's context. That makes the CLI the better fit for anything an agent runs unattended.

Authoring off the merge path

Authoring is where agents save the most time, and where unreviewed output does the most damage. Keep all of it on a branch or in a PR.

Agent definitions

Add the agent definitions with init-agents and commit them like any other config. The docs say to regenerate them whenever Playwright is updated, so they pick up new tools and instructions. Make that part of every upgrade PR:

# Run once on setup, then again in every PR that upgrades Playwright
npm install --save-dev @playwright/test@latest
npx playwright init-agents

The seed test

The planner runs the seed test to do all the initialization your tests need: global setup, project dependencies, fixtures and hooks. A seed that's a bare page.goto hands the planner a logged-out page with none of your data, and the plan it writes describes an app your suite never sees. Build the seed on the fixtures the suite already uses:

// tests/fixtures.ts
import { test as base, expect, type Page } from '@playwright/test';

export const test = base.extend<{ signedInPage: Page }>({
  signedInPage: async ({ page }, use) => {
    await page.goto('/login'); // needs use.baseURL in playwright.config.ts
    // Credentials come from the environment, never from a plan or prompt
    await page.getByLabel('Email').fill(process.env.E2E_USER_EMAIL!);
    await page.getByLabel('Password').fill(process.env.E2E_USER_PASSWORD!);
    await page.getByRole('button', { name: 'Sign in' }).click();
    await expect(page.getByRole('heading', { name: 'Dashboard' })).toBeVisible();
    await use(page);
  },
});

export { expect };

// tests/seed.spec.ts
import { test, expect } from './fixtures';

test('seed', async ({ signedInPage }) => {
  await expect(signedInPage.getByRole('heading', { name: 'Dashboard' })).toBeVisible();
});

The fixture reads credentials from environment variables instead of hard-coding them in the spec or the seed.

The plan as contract

The planner's Markdown plan is the cheapest place to catch a wrong test, so review it before the generator writes any code. Make it state what a reviewer needs to check: the intent of each scenario, the business rules behind it ("the cheapest fare is preselected", "a trial user can't export"), and the negative and boundary cases, such as an expired card or a cart at its item limit. Plain language works here because the people who know the rules can read it. The generated code gets checked against the plan, so a vague plan gives you code nobody can check.

Generated specs land as PRs

Generated tests can start with errors, so nothing the generator writes goes straight to main. Its specs arrive as a pull request and get reviewed like any other code. Reviewers who want a list to work from can use our pre-merge checklist for AI-generated Playwright tests. Exploration follows the same rule: an agent bug bash on a branch ends in a plan or an issue, not in a commit to main.

The deterministic merge gate

The merge gate runs committed spec files and nothing else. Agents propose code in branches and PRs, and they never run inside the gating job.

Playwright controls this with the --run-agents option, which runs agents to generate the code for page.perform. The option accepts missing, all or none, and none is the default.

# Anti-pattern: agents generate page.perform code on every gating run,
# so one commit can take different paths and get different verdicts
name: PR gate
on: pull_request
jobs:
  e2e:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: npm ci
      - run: npx playwright install --with-deps
      - run: npx playwright test --run-agents=all
        env:
          AGENT_API_KEY: ${{ secrets.AGENT_API_KEY }}  # placeholder: the credential your agent's model provider needs

name: PR gate
on: pull_request
jobs:
  gate:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: npm ci
      - run: npx playwright install --with-deps
      # Committed specs only: no model runs in the gate
      - run: npx playwright test --run-agents=none
  repeat-agent-specs:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: npm ci
      - run: npx playwright install --with-deps
      - name: Repeat new or changed agent-written specs
        run: |
          CHANGED=$(git diff --name-only --diff-filter=AM "origin/${{ github.base_ref }}...HEAD" -- 'tests/agent/*.spec.ts')
          if [ -z "$CHANGED" ]; then
            echo "No agent-written specs changed"
            exit 0
          fi
          npx playwright test $CHANGED --run-agents=none --repeat-each=10

Generated code as frozen replay

By default, agents don't run to generate the code for page.perform: the --run-agents option defaults to none. Committed generated code changes only when someone regenerates it, and the regenerated diff goes through review again like any other change.

Assertions as postconditions

Follow every agent-generated step with exact expect assertions. They are postconditions the agent can't edit at runtime: whatever route the generated code took, the test still has to end on the state you wrote down. A step that reached the wrong state fails right there, on the assertion.

Repair agents on CI failures

A repair agent is only as good as the evidence it starts from. Trigger it from a real CI failure, and give it more than the log. A CI log doesn't show what was on the page, which request returned the wrong data, or whether the test failed the same way last week. An agent working from the log alone can't see any of that, so it guesses, which is why a coding agent needs the failing run, not just the log.

import { test, expect } from '@playwright/test';

// Anti-pattern: the "repair" waits out the symptom and accepts any total,
// so the test now passes against the pricing bug it caught
test('order total includes shipping', async ({ page }) => {
  // assumes use: { baseURL } is set in playwright.config.ts
  await page.goto('/checkout');
  await page.getByRole('button', { name: 'Place order' }).click();
  await page.waitForTimeout(5000); // added by the agent
  // was: await expect(page.getByTestId('order-total')).toHaveText('$54.00');
  await expect(page.getByTestId('order-total')).toContainText('$');
});

name: PR gate
on: pull_request
jobs:
  gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: npm ci
      - run: npx playwright install --with-deps
      - run: npx playwright test --run-agents=none
      - uses: actions/upload-artifact@v4
        if: failure()
        with:
          name: test-results
          path: test-results/
  repair:
    needs: gate
    if: failure()
    runs-on: ubuntu-latest
    permissions:
      contents: write
      pull-requests: write
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: npm ci
      - run: npx playwright install --with-deps
      - uses: actions/download-artifact@v4
        with:
          name: test-results
          path: test-results/
      - name: Ask the coding agent for a classification and a draft PR
        run: ./scripts/repair-agent.sh
        env:
          AGENT_RUN: '1'
          AGENT_API_KEY: ${{ secrets.AGENT_API_KEY }}
          REPAIR_PROMPT: >-
            A Playwright test failed in CI. Its traces are in test-results/.
            Explore them with npx playwright trace, and attach to the failing
            test with xvfb-run npx playwright test --debug=cli. First classify the failure
            as regression, stale test, environment or flake. Change code only for
            a stale test, never by adding a wait or loosening an assertion, and
            open the change as a draft pull request. For anything else, report
            the classification and the evidence without a fix.

When the healer skips

The healer re-runs a test until it passes or until guardrails stop the loop, and it returns either a passing test or a skipped test when it believes the functionality is broken. Treat that skip as a bug report, and never merge one automatically. A merged skip removes the check for exactly the feature the healer thinks is broken.

Classification stays with you and your own agent, not the healer. A feature request asked for a triage agent that runs before the healer, because the reporter worried that when the app really regressed, the healer could leave a test quietly adjusted to match the bug (#43675). The same reply added that you are probably better off asking your agent to fix the tests than using the healer at all.

Time and token budgets

An agent loop has no natural end, so the pipeline has to supply one. Playwright won't by default: globalTimeout, the maximum time in milliseconds the whole suite can run, defaults to zero, and zero disables it. Set it on every run an agent starts, and back it with a timeout on the CI job.

The per-test timeout needs the same attention. Playwright Test enforces 30 seconds per test by default, and that budget includes fixture setup and beforeEach hooks along with the test function. An agent-written test that logs in through a fixture and passes on a fast laptop can run out of that budget on a slower CI runner. Make both limits explicit:

import { defineConfig } from '@playwright/test';

// Set AGENT_RUN=1 in every job an agent starts
const agentRun = !!process.env.AGENT_RUN;

export default defineConfig({
  testDir: './tests',
  // Zero, the default, means no limit; agent runs get a hard ceiling
  globalTimeout: agentRun ? 15 * 60 * 1000 : 0,
  // Covers the test function, fixture setup and beforeEach hooks
  timeout: 30 * 1000,
});

Fifteen minutes is a placeholder: pick the ceiling from how long your suite normally takes.

The healer's own guardrails are not your CI budget. Cap attempts per failure and per run in whatever script starts the agent, and stop at the cap.

Every agent step also costs something on each run: time, tokens, variance in how long the run takes, and a failure that is harder to explain. Budget agent runs the way you budget CI minutes, per run and per PR.

Where tokens go

Snapshots and traces are where an agent's context fills up fastest. Hand the agent the failing test, its error and the one artifact that explains it, never whole directories.

Idle browsers

A headless browser launched by the Playwright MCP server closes after an hour without tool calls. An agent session that pauses longer than that, waiting on a human review for instance, loses the browser, and the next tool call launches a new one, so the agent has to navigate again.

Debugging agent runs with traces

When an agent-written test fails, the trace is the evidence, for you and for the agent that tries to repair it. Keep it. With trace: 'retain-on-failure', Playwright records a trace for every run but keeps only the ones that failed, and it keeps a failed run's trace even when a later retry passes. The attempt that failed is the one worth explaining. A person reading it can follow our step-by-step trace viewer walkthrough for red CI jobs.

Two commands give a coding agent its own way in. npx playwright trace lets it explore a trace from the command line to understand a failing or flaky test. npx playwright test --debug=cli lets it attach to and debug tests over playwright-cli, which the release notes call perfect for automatically fixing tests in agentic workflows:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  use: {
    // Keep a failed attempt's trace even when a later retry passes
    trace: 'retain-on-failure',
  },
});

# Open the failing test's trace, then list only its assertion actions
npx playwright trace open test-results/<test-dir>/trace.zip
npx playwright trace actions --grep="expect"
# Attach to the failing test while fixing it
npx playwright test tests/agent/checkout.spec.ts --debug=cli

Judging agent-written tests

Judge agent-written specs on the same three numbers as the specs your team wrote: flake rate, retry count and duration. Specs that flake more, retry more or run longer cost more than they save, however fast an agent wrote them.

import { defineConfig } from '@playwright/test';

export default defineConfig({
  testDir: './tests',
  // Off by default; with retries on, a fail-then-pass is reported as flaky
  retries: process.env.CI ? 2 : 0,
  // Exit with an error when any test is marked flaky
  failOnFlakyTests: !!process.env.CI,
  use: { trace: 'retain-on-failure' },
});

CI=1 npx playwright test tests/agent --repeat-each=20
CI=1 npx playwright test tests/human --repeat-each=20

Isolated retries

When a spec flakes only inside the full suite, try the 'isolated' retry strategy. It runs retries at the end, after all other tests finish, one by one in a single worker, which cuts interference between retried tests and the rest of the suite at the cost of a longer run. A spec that still flakes in isolation is flaky on its own. One that passes there was colliding with something else in the suite.

Before generating more

Keep agent-written specs in their own directory while you roll agents in, so the numbers stay separate.

Then look past the green runs. Count how often the specs break on a UI state the plan didn't cover, or on a page object the generator didn't use. Check that every failure leaves a trace that explains it. Add the tokens and review time each spec took to produce to its run duration. Last, watch whether the team acts on a red result from these specs without re-running it first, because a result people re-run before they believe it hasn't earned their trust.

Settle each of those questions with evidence you can check yourself, starting with Playwright's own pass, fail and flaky results, and never with a second model reviewing the first.

When not to use agents

Keep the core regression gate fully specified. When a check has to give the same verdict on every commit, an agent adds cost and variance and nothing else.

Don't hand a real regression to the healer. Its skipped test is the bug report, as the repair section showed.

Prefer exact assertions to an agent's judgment. An agent that writes assertions from what it sees can freeze today's bug into the test: if the total is wrong when the agent looks, the wrong total becomes the expected value. A natural-language assertion checked by a model has the same weakness, since it passes whatever the model accepts.

Stop adding agents where a step's token and review cost is higher than the time it saves. The gate is where that happens first. Stop generating specs for an area, too, while its agent-written specs flake, retry or run longer than the ones your team wrote.

Frequently asked questions