AI Playwright Tests: MCP, CLI, Agents and Prompts Compared

Playwright comes with three test agents out of the box: planner, generator and healer. A test an agent writes passes in the chat session, in a browser window you watched, on your laptop. That pass tells you little. The test still has to survive headless parallel workers, a shard split and repeated runs in CI, next to the rest of your suite.

The code needs work too. Without guidance, agents write Playwright tests from generic patterns: waitForTimeout calls, brittle CSS selectors and retries that hide real bugs. A generated test that also scripts its own login, or builds page objects next to the ones you already have, ignores the setup your suite depends on. Someone then has to strip all of that out in review.

Ranking the methods in theory, by how much context each gives the model, won't tell you which one writes tests you can keep. You find out by running them on the same scenarios, in your own suite, through the same gate. That gives you a scorecard for picking a method per job, and a stability gate that every AI-written test has to clear before merge.

Four ways to get an AI to write a Playwright test, and the context each one sees

"Playwright AI" is not one product, and treating it as one is the first mistake. When a generated test fails, "the AI got it wrong" gives you nothing to fix. Playwright CLI gives the model concise CLI output and loads skills on demand, while MCP puts tool schemas and snapshots in the model's context. A prompt-only assistant gets the least context of all.

• Playwright MCP. The Playwright MCP server lets an LLM interact with web pages through structured accessibility snapshots over the Model Context Protocol. Playwright MCP operates on the page's accessibility tree, not pixels.
• Playwright CLI. Playwright CLI is a command-line interface for browser automation designed for coding agents, with token-efficient commands meant for limited context windows. After each command, playwright-cli outputs a snapshot of the current browser state: an accessibility tree with element refs.
• Planner, generator and healer. Playwright ships these three test agents out of the box. These agents can be used independently, sequentially, or as the chained calls in the agentic loop.
• A prompt-only coding assistant. ChatGPT in a tab, or an editor assistant with no browser connected, given the scenario in plain language. It may see your repo, but it never sees a live page. It is the baseline the other three have to beat.

Recording a test by hand, with no model in the loop, is a different job and stays out of this comparison.

Context decides the quality of a generated test more than the model does. Your source code shows the model a button component. It doesn't show which buttons render on the team settings page for an admin, what the confirmation says after the form submits, or which cookies the app set along the way. A Playwright MCP snapshot shows the page elements, their roles and their text content. So hold the model and the scenarios fixed, and vary only the context.

Build the bench: a seed test, Markdown scenarios and the suite's own login

If each method gets its own description of the scenario and logs in its own way, you end up grading prompts, not methods. The bench fixes three inputs: the scenario, the login and the seed test.

One Markdown scenario per flow

Write each scenario once, as a Markdown file in the repo, and give every method that same file. A scenario can start life as a ticket or a paragraph in a PRD. Once you commit it, every method reads the same file, so a difference in output comes from the method rather than from how you phrased the request that day.

# Invite a teammate

Start: signed in as an admin, on the Team settings page.

1. Click "Invite member".
2. Fill "Email address" with an address no other test uses.
3. Choose the role "Editor" and click "Send invite".

Expected:
- A message says "Invite sent".
- The new address appears under "Pending invites" with the role "Editor".

Write every expected result as something a test can assert. Then "did the method cover the scenario" becomes a yes-or-no question: each line under "Expected" has an assertion or it doesn't.

Login comes from a setup project

The anti-pattern is letting each method invent its own login. You get four tests that each fill in the login form, and we'd expect all four to break the day someone changes the login page. Tests that load existing authenticated state don't need to authenticate in every test, and Playwright's docs say that speeds up test execution.

Playwright's auth docs recommend a setup project that every testing project depends on. The setup project always runs and authenticates before all the tests, and the testing projects use the authenticated state as storageState. Generated tests then start signed in and need no login step:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  testDir: './tests',
  use: {
    baseURL: process.env.BASE_URL,
  },
  projects: [
    { name: 'setup', testMatch: /.*\.setup\.ts/ },
    {
      name: 'chromium',
      use: {
        browserName: 'chromium',
        // Every test in this project starts from the state the setup project saved
        storageState: 'playwright/.auth/admin.json',
      },
      dependencies: ['setup'],
    },
  ],
});

The setup file signs in once and saves the state the testing project loads:

// tests/auth.setup.ts
import { test as setup, expect } from '@playwright/test';

const authFile = 'playwright/.auth/admin.json';

setup('authenticate as admin', async ({ page }) => {
  await page.goto('/login');
  await page.getByLabel('Email').fill(process.env.ADMIN_EMAIL!);
  await page.getByLabel('Password').fill(process.env.ADMIN_PASSWORD!);
  await page.getByRole('button', { name: 'Sign in' }).click();
  await expect(page.getByRole('heading', { name: 'Dashboard' })).toBeVisible();
  await page.context().storageState({ path: authFile });
});

That single admin account has a limit. Inviting a teammate changes server-side state, and for tests that modify server-side state, the auth docs recommend authenticating each parallel worker once, with one account per worker (our auth guide shows how to isolate auth per worker). Without per-worker accounts, generated tests that pass one at a time will collide once they run in parallel workers.

A seed test that looks like your best test

The seed test matters more than any prompt. The planner runs the seed test to execute the suite's initialization: global setup, project dependencies, fixtures and hooks. The planner also uses the seed as the example for every test it generates. Give the seed your real fixtures and page objects, and hand the same file to the other three methods as the example to copy.

The suite's fixtures file exposes a page object for the team settings page:

// tests/fixtures.ts
import { test as base, expect, type Page } from '@playwright/test';

export class TeamSettingsPage {
  constructor(private readonly page: Page) {}

  async goto() {
    await this.page.goto('/settings/team');
  }

  async invite(email: string, role: string) {
    await this.page.getByRole('button', { name: 'Invite member' }).click();
    await this.page.getByLabel('Email address').fill(email);
    await this.page.getByLabel('Role').selectOption(role);
    await this.page.getByRole('button', { name: 'Send invite' }).click();
  }
}

export const test = base.extend<{ teamSettings: TeamSettingsPage }>({
  teamSettings: async ({ page }, use) => {
    await use(new TeamSettingsPage(page));
  },
});

export { expect };

The seed test uses that page object the way your best existing test would:

// tests/seed.spec.ts
import { test, expect } from './fixtures';

test('seed: team settings as a signed-in admin', async ({ page, teamSettings }) => {
  await teamSettings.goto();
  await expect(page.getByRole('heading', { name: 'Team settings' })).toBeVisible();
});

The opposite holds too. A seed built from raw CSS locators and an inline login teaches every generated test to do the same.

Run Playwright MCP against the scenarios

MCP is the method most likely to hand you a test that passes for the wrong reason. Most of the fix is in the prompt, and the rest is in what you refuse to believe afterwards.

Give the MCP-connected agent a prompt in three parts: the environment, the scenario, and instructions to explore with snapshots first and write the test second. Keep the prompt in a file next to the scenario, so every run uses the same words:

## Environment
- App: http://localhost:3000
- The MCP server runs with --isolated --storage-state=playwright/.auth/admin.json, so the browser is signed in as the admin test user that the setup project uses.

## Scenario
Follow specs/invite-teammate.md step by step.

## Instructions
1. Explore first. Take a snapshot before every interaction and act only on elements in it.
2. Use only elements a user can see and reach in this flow.
3. Then write tests/generated/invite-teammate.mcp.spec.ts:
   - import { test, expect } from '../fixtures' and use the teamSettings page object
   - rely on the storageState from the setup project; never script a login
   - use role- and label-based locators; no CSS selectors, no waitForTimeout
   - add one assertion for every line under "Expected"
4. Run the test, fix it, and stop when it passes.

Two defaults decide what the green run during generation is worth. By default, Playwright MCP runs the browser in headed mode, so a pass in that visible browser says nothing yet about a headless CI run. Playwright MCP also uses a persistent profile by default, so login state and cookies are preserved between sessions (see how the MCP server manages state and sessions).

The anti-pattern is accepting the MCP test because it passed in that session. The profile may still hold yesterday's login state and cookies. A test written against that browser can pass there for reasons the test never set up. In CI, the testing project runs it with the storageState your setup project writes, not with the MCP browser's profile, so the gate is where you find out.

Read every interaction against the scenario, too. In a Playwright issue, a user reported that the MCP accessibility tree snapshot included off-screen, non-viewport elements, causing AI agents to generate invalid test cases on unreachable UI elements. If the test touches something the scenario never mentions, find out why before you keep it.

Iterate in the same session. When the model guesses, fix the scenario file rather than the test, regenerate, and stop when every expected result has an assertion. Then the test goes to the gate.

Run Playwright CLI snapshots and keep the code each command prints

MCP keeps tool schemas and snapshots in the model's context, so its token cost is higher. Playwright CLI is a command-line interface for browser automation designed for coding agents: token-efficient commands and installable skills let agents balance browser automation with large codebases and reasoning within limited context windows. In the docs' CLI-vs-MCP comparison, the CLI's token cost is rated lower because its output is concise and skills load on demand, and MCP's is rated higher because tool schemas and snapshots sit in context.

After each command, playwright-cli outputs a snapshot of the current browser state: an accessibility tree with element refs the agent uses for the next interaction. With the browser signed in as the bench's test account, an agent working the invite scenario runs commands like these, taking each ref from the latest snapshot:

playwright-cli open http://localhost:3000/settings/team
playwright-cli click e12
playwright-cli fill e31 "new.member@example.com"
playwright-cli click e36
playwright-cli snapshot

The snapshot after open contains nodes like these (abridged):

- heading "Team settings" [level=1]
- button "Invite member" [ref=e12]
- region "Pending invites"

The anti-pattern is reusing a ref from a snapshot taken before the page changed. Refs are stable within one snapshot but invalidated when the page changes. The docs' rule is to re-snapshot after navigation, which is why the session ends with snapshot after the invite form submits.

The refs are throwaway; the code is not. Every playwright-cli action prints the Playwright code it just ran, and the docs call that code the raw material for tests. For the first click and the fill above, the printed code reads:

await page.getByRole('button', { name: 'Invite member' }).click();
await page.getByRole('textbox', { name: 'Email address' }).fill('new.member@example.com');

The generated code prefers role-based locators, which the docs say survive markup changes far better than CSS selectors. Keep those lines. Have the agent build the test from them, swap raw page calls for the suite's page object where one exists, and add an assertion for each expected result.

When a generated test fails, run it with --debug=cli in the background. The test stays paused and holds the browser open while the agent works:

npx playwright test tests/generated/invite-teammate.cli.spec.ts --debug=cli &

Inspecting the paused page beats asking the model to guess from a stack trace.

Playwright issue 39847 proposes AI-powered test generation from codegen, with network capture and prompt export. In issue 39847, a user proposed having codegen record a human's actions and export them as a prompt for any AI assistant. Playwright CLI is a command-line interface for browser automation designed for coding agents. For a bench that compares what agents produce, the agent-driven side is the one to test.

Run the Planner, Generator and Healer agents

The test agents come with a repair loop built in: the healer executes the test suite and automatically repairs failing tests. That repair loop is the part that needs a human watching.

Add the agent definitions to the project with the init-agents command, and regenerate them whenever you update Playwright, so the agents pick up new tools and instructions. The bench runs it with the VS Code loop:

npx playwright init-agents --loop=vscode

The agents work from the files the bench already has:

specs/invite-teammate.md     the scenario; we keep the planner's plan beside it
tests/fixtures.ts            page objects and fixtures
tests/seed.spec.ts           the seed the planner runs and copies
tests/generated/             generated tests, one file per method and scenario

Point the planner at the scenario and the seed. The planner runs the seed test first, so it starts from your real setup, and then writes a Markdown plan. Read the plan before the generator touches it: a wrong step in the plan becomes a wrong test. The generator uses the Markdown plan to produce executable Playwright tests, and it verifies selectors and assertions live as it performs the scenarios.

Generated tests may still contain initial errors, and the healer agent can fix those automatically. The healer re-runs the test until it passes or until guardrails stop the loop. Its output is either a passing test, or a skipped test when it believes the functionality itself is broken.

The anti-pattern is merging the healer's output unread. A passing test tells you the loop ended, not that the test still checks what the scenario asked for, so check that every healer diff is additive, not subtractive, the way you'd review a colleague's change. A skip is a finding, not a fix. On the bench, report a skipped test as a possible regression and check the feature by hand before anyone counts the test as healed.

Three reports in the Playwright repo show where the loop rubs:

• In issue 37735, a user asked for the agents to use a suite's existing page objects instead of generating masses of duplicate test code.
• In issue 38068, a user reported that the generator only printed tests in chat, saying it didn't have file editing tools enabled. The same user reported in issue 38068 that the healer agent could create folders and files. Issue 38068 lists initializing the agents with the VS Code loop as a step to reproduce.
• In issue 37749, a user reported that the seed file ran fine when run directly but failed when the Planner agent used it as context. Issue 37749 lists initializing the agents with the VS Code loop as a step to reproduce.

Each of those lands on the scorecard: duplicated page objects as reviewer edits, chat-only output as extra time.

Run a prompt-only coding assistant as the baseline

Ask a coding assistant for a test from a plain-language scenario, with no live page, and the model writes from what Playwright tests usually look like, not from your app. Pasting source code into the context doesn't close the gap. Components show what could render, not what the page renders for this user with this data.

The baseline output to grade looks like this anti-pattern:

// Anti-pattern: hard waits, CSS selectors and a scripted UI login
import { test, expect } from '@playwright/test';

test('invite a teammate', async ({ page }) => {
  await page.goto('http://localhost:3000/login');
  await page.fill('#email', 'admin@example.com');
  await page.fill('#password', 'password123');
  await page.click('button.btn-primary');
  await page.waitForTimeout(3000);
  await page.goto('http://localhost:3000/settings/team');
  await page.click('.invite-btn');
  await page.fill('input[name="email"]', 'new.member@example.com');
  await page.selectOption('select.role', 'Editor');
  await page.click('.modal-footer .btn-primary');
  await page.waitForTimeout(2000);
  await expect(page.locator('.pending-invites tr').last()).toContainText('new.member@example.com');
});

Every problem in it shows up later. The two hard waits are guesses tuned to one machine. The CSS classes are tied to markup, so the next restyle of the modal breaks them. The login duplicates the setup project and breaks with the login page. And --repeat-each runs each test N times (default: 1), so with N above 1 every run invites the same fixed address, which the scenario rules out.

Playwright's best practices recommend locators that prioritize user-facing attributes and explicit contracts. Here is the same scenario written against the bench:

// tests/generated/invite-teammate.prompt.spec.ts
import { test, expect } from '../fixtures';

test('admin can invite a teammate as editor', async ({ page, teamSettings }) => {
  // A unique address per run, so repeats and parallel workers don't collide
  const email = `invite-${Date.now()}-${Math.random().toString(36).slice(2, 8)}@example.com`;

  await teamSettings.goto();
  await teamSettings.invite(email, 'Editor');

  await expect(page.getByText('Invite sent')).toBeVisible();
  const pendingInvites = page.getByRole('region', { name: 'Pending invites' });
  await expect(pendingInvites.getByRole('row', { name: email })).toContainText('Editor');
});

The test starts signed in because the config gives it the setup project's storageState. It reuses the page object, and every locator in the test body names something a user sees.

Correcting the assistant toward that test works once. Writing the same corrections into every prompt doesn't scale. Currents' Playwright best practices skill works in Claude Code, Cursor, VS Code, Gemini CLI and other agents that read skills, loads on its own when a task involves Playwright, and covers the Page Object Model, fixtures, and global setup and teardown. If you keep your own shared prompt file instead, it should name the suite's fixtures file, forbid hard waits and CSS selectors, and require an assertion for every expected result.

Run the baseline twice, with the guidance loaded and without it, so the scorecard shows what guidance adds.

Score every generated test on the same rubric

Ranking methods by first-run pass rate alone rewards the wrong tests: the ones that pass first time because the MCP profile was already signed in, or because a hard wait happened to be long enough. Record all of these for every method and scenario instead:

Grade locators against Playwright's best practices, and count a CSS selector the reviewer replaces as an edit even when the test passed with it.

Reuse is the easiest measure to check. Every method got the seed test as its example, so a test that imports from @playwright/test directly, builds its own page object or signs in through the form counts against that method.

Reviewer edits capture what a pass hides: simplifying an overcomplicated script, renaming vague steps, and fixing an assertion that checks something the spec never asked for.

Measure time and tokens yourself. The docs' comparison says which way the CLI and MCP lean on token cost, but your scenarios decide the real numbers, so log them per scenario. A method that saves tokens but needs a second round of review costs more than it saved.

Weight stability and reviewer edits above first-run pass. A test that passes once and flakes later costs more than one that needed a single fix before merge, because the flaky one fails in someone else's PR.

Gate generated tests in headless CI: workers, shards and --repeat-each

A generated test earns its merge under the conditions CI imposes, not the ones it was written under. By default, Playwright runs test files in parallel, and tests in a single file run in order in the same worker process. When a generated test relies on what an earlier test in the same file left behind, it passes in that one worker. Move it into its own file, and it fails the first time it lands on a different worker.

The anti-pattern is a single green run on the machine that generated the test:

# Anti-pattern: one run, with whatever worker count this machine defaults to
npx playwright test tests/generated/invite-teammate.mcp.spec.ts

The run that counts repeats every generated test under a fixed worker count:

npx playwright test tests/generated --repeat-each=10 --workers=2

--repeat-each runs each test N times, and its default is 1, so without the flag a flaky test gets one chance to look fine. --workers takes a count or a percentage of logical CPU cores and defaults to 50%. When your laptop and the CI runner have different core counts, the same command runs with different parallelism on each. Fix the count so a failure on one machine reproduces on the other. State-changing tests that share one account show up here too: they pass alone and collide under parallel workers, which is the case for one account per worker.

--shard runs only the selected shard, given as a 1-based current/all value such as 3/5. Run the generated tests across shards, alongside the rest of the suite, with the same split between shards and workers your CI already uses:

name: Generated tests gate

on:
  pull_request:
    paths:
      - 'tests/generated/**'

jobs:
  gate:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shardIndex: [1, 2, 3, 4]
        shardTotal: [4]
    env:
      BASE_URL: ${{ vars.BASE_URL }}
      ADMIN_EMAIL: ${{ secrets.ADMIN_EMAIL }}
      ADMIN_PASSWORD: ${{ secrets.ADMIN_PASSWORD }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: lts/*
      - run: npm ci
      - run: npx playwright install --with-deps
      - name: Run the suite on this shard
        run: npx playwright test --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }} --workers=2
      - name: Repeat the generated tests on this shard
        run: npx playwright test tests/generated --repeat-each=10 --workers=2 --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}

The first test step runs the whole suite on its shard, generated tests included, with nothing repeated. The second step repeats only the tests in files under tests/generated.

The Generator agent produces executable Playwright tests, so the gate runs them with npx playwright test like any other test file.

Which method to use for which job, and when to write the test yourself

No method wins every scenario, and picking one for everything because it won a single comparison throws away what the bench told you. Use each where its context fits:

• Playwright MCP for exploration and single tests in an editor chat, where you can watch the browser and refine the scenario as you go.
• Playwright CLI when long agent sessions would fill the context window, or when you want the printed code as the raw material for the test.
• The test agents when you want a plan-to-test pipeline and have a seed test good enough to copy. Keep a human on every healer change and every skip.
• Prompt-only generation only with guidance loaded, and expect the most reviewer edits.

Write the test yourself when the scenario depends on data the live page doesn't show, such as an email the app sends or a record another system writes. Do the same when the agent writes duplicate test code instead of reusing your page objects, the problem one user raised in issue 37735. Fighting the generator past that point costs more than writing the test.

The generator agent produces executable Playwright tests. Cheaper test creation moves the bottleneck to review and stability, so the gate matters more than which generator you pick.

After merge: watch whether AI-written tests stay stable

The pre-merge gate catches flakiness you can reproduce on demand. Flakiness that only appears across weeks of CI runs, on data and timing the gate never saw, needs tracking per test. Currents Test Explorer shows flaky, failing and long-running tests and the trends in their behavior, with Flakiness Rate as the percentage of executions whose outcome is non-deterministic.

Keep the method in every generated test's file name, as the bench does with invite-teammate.mcp.spec.ts, so Test Explorer's search by spec name can narrow the results to one method. The scorecard then keeps updating from production runs instead of one bench run. If MCP-written tests start flaking a month after merge and agent-written ones don't, that gap is the comparison that counts, and it should decide which method gets the next scenario.