Reviewing AI-Generated Playwright Tests Before They Reach CI

The Generator finishes, and you get a spec file that passes on your machine. Someone opens a pull request. Now you have to decide whether it belongs in a suite that gates deployments.

The Playwright documentation tells you to review generated code, and so does every tutorial written since the agents shipped. None of them defines what that review looks like. The checks you already use for human-written tests were built for a different set of mistakes.

That calibration gap is the problem this article solves. Review generated tests for the mistakes agents make, rather than the ones people make.

When a colleague writes a test, you check the locator for sense, the assertion for meaning, and the setup for whether it belongs in a fixture. Those checks still matter. They were designed around how people make mistakes, though, and an agent makes a different set.

The seven checks ahead map onto three failure modes, and knowing the mapping tells you what to look for as you read. Checks 1 and 2 cover what the test claims to verify. Checks 3 through 5 cover how it reaches the state it tests against. Checks 6 and 7 cover what the Healer, the agent that repairs failing tests without asking you first, changed on its way to green.

Hallucinated assertions. The agent invented a verification step that appears valid but does not meet your acceptance criteria. It asserts what it expects the interface to do rather than what it was told to verify, so the test passes even though it checks the wrong thing.

Intent-weakening heals. The Healer repaired a broken test by broadening a locator or rewriting an assertion, leaving the test less specific than it started, and often pointing at a different element than the original. The code compiles, the test passes, and the failure it was written to catch now goes unreported.

UI-driven state setup. The agent built state by clicking through the UI rather than using API fixtures or storageState. The agent has no other way to reach a logged-in state, so during generation, this is correct behavior. It is the wrong approach for CI, where every setup step is another source of flakiness.

You get seven checks that catch these before merge, with runnable code for each one. Checks 1 through 5 apply to any source of generated Playwright code, whether from Playwright Test Agents or MCP-driven generation through Claude Code or Cursor. Checks 6 and 7 are specific to the Healer, the automated repair agent that ships with Playwright Test Agents. If you generate tests through MCP without the Healer loop, skip to the checklist summary after check 5.

For background on the agents themselves, Currents has written on getting the most out of Playwright Test Agents and on the state of the Playwright AI ecosystem in 2026.

The three failure modes agents introduce that humans do not

Start with the mechanism behind each one. A reviewer who knows why a mistake happens spots it faster than one working from a list.

What makes these three modes specific to agents is that each traces to a constraint under which no human author works. Acceptance criteria are invisible to it. Success is defined as making the test pass rather than catching bugs, and the browser is the only interface it has. A human author writing a bad test still knows what the feature is supposed to do. An agent never does.

Failure mode 1: hallucinated assertions

The Generator writes verification steps from what it observed in the browser. It performs each step live through MCP tools, watches what changes on the page, and turns that observation into an assertion. Nothing in that loop connects to your acceptance criteria.

An agent can interact with a Submit Order button, and it cannot determine whether the correct backend side effects occurred unless those checks are defined for it. The agent asserts what it could see. The order record you care about lives in a database it never queried.

Here is a checkout page with a real bug. The URL updates, the confirmation heading renders, and the order number is never populated.

<!doctype html>
<html>
  <body>
    <h1>Cart</h1>
    <button id="checkout">Checkout</button>
    <div id="confirm" style="display:none">
      <h2>Order confirmed</h2>
      <span data-testid="order-number"></span>
    </div>
    <script>
      document.getElementById('checkout').addEventListener('click', () => {
        history.pushState({}, '', '/confirmation');
        document.getElementById('confirm').style.display = 'block';
        // order-number is never filled: the order was not recorded
      });
    </script>
  </body>
</html>

The generated test asserts the navigation, because navigation is what the agent watched happen.

test('generated: checkout redirects to confirmation', async ({ page }) => {
  await page.goto('/app.html');
  await page.getByRole('button', { name: 'Checkout' }).click();
  await expect(page).toHaveURL(/\/confirmation/);
});

That test passes. The reviewed version asserts the outcome a customer would care about.

test('reviewed: checkout records an order number', async ({ page }) => {
  await page.goto('/app.html');
  await page.getByRole('button', { name: 'Checkout' }).click();
  await expect(page.getByRole('heading', { name: 'Order confirmed' })).toBeVisible();
  await expect(page.getByTestId('order-number')).not.toBeEmpty();
});

That one fails, as it should, because the order was never recorded. Running both against the page above on Playwright 1.56 gives one pass, and one failure, and the passing test is the one that tells you nothing.

The detection rule follows from the mechanism. An assertion that checks only a navigation event, a URL pattern, or a page title without checking that the business outcome appeared is a candidate for rewriting. Navigation is the easiest thing for an agent to observe, which makes it the most likely thing for an agent to assert.

Observation drives the second failure mode, too, with a different agent and higher stakes, because the Healer has permission to change existing code.

Failure mode 2: intent-weakening heals

The Healer has a harder problem than the Generator. Its instructions make the risk concrete. Run npx playwright init-agents --loop=claude and open the generated playwright-test-healer agent definition (you can also read the source on GitHub). The definition tells the agent to update selectors, fix assertions and expected values, and re-run until the test passes cleanly. It also says this:

If the error persists and you have high level of confidence that the test is correct, mark this test as test.fixme() so that it is skipped during the execution.

And this:

Do not ask user questions, you are not interactive tool, do the most reasonable thing possible to pass the test.

Read those two lines next to each other. You have an agent with permission to edit assertions, permission to skip tests, no permission to ask you anything, and a success condition defined as the test passing. Most of the time that produces a correct locator update. When it does not, the failure is silent, because the output of a bad heal looks the same as the output of a good one.

Take a button renamed from "Submit Order" to "Place Order" during a refactor, on a page where the button is also disabled by a genuine bug.

<button id="cancel">Cancel</button>
<button id="place" disabled>Place Order</button>

Three versions of the same check. The original fails on the old name. The weakened heal broadens the locator until it cannot fail. The correct heal keeps the specificity and updates the name.

// Original: fails, the button was renamed
await expect(page.getByRole('button', { name: 'Submit Order' })).toBeEnabled();

// Weakened heal: matches Cancel, passes, verifies nothing
await expect(page.getByRole('button').first()).toBeEnabled();

// Correct heal: same specificity, new name, correctly fails
await expect(page.getByRole('button', { name: 'Place Order' })).toBeEnabled();

Run those three, and the middle one is the only test that passes. It passes because it is checking a different element. .first() resolves to Cancel, Cancel is enabled, and the assertion succeeds while the real Place Order button sits disabled and unexamined. The heal did not make the assertion weaker in the abstract. It repointed the assertion at a button nobody asked about, which is the precise thing to look for when reading a heal diff.

Three more rewrites produce a check that cannot fail, and each looks reasonable in isolation.

// Passes because nothing named Ghost exists
await expect(page.getByRole('button', { name: 'Ghost' })).not.toBeVisible();

// Passes for the same reason
await expect(page.getByRole('button', { name: 'Ghost' })).toHaveCount(0);

// Passes against whichever button happens to come first
await expect(page.getByRole('button').first()).toBeVisible();

Each of these passes because the locator resolves to the wrong thing or to nothing at all. not.toBeVisible() and toHaveCount(0) both succeed against a locator matching zero elements, since zero elements are trivially not visible and trivially number zero. toBeHidden() has the same behavior. Its contract is "locator either does not resolve to any DOM node, or resolves to a non-visible one," so it passes on zero matches too. Verified on 1.56, all three pass against a page where the named button does not exist.

The defense is a two-step assertion: first confirm the element exists, then check its state. await expect(locator).toHaveCount(1) before the state check catches a locator that silently matches nothing.

That is the shape to look for in a heal diff. Be precise about which rewrites carry risk. Swapping toBeEnabled() for not.toBeDisabled() does not. Both fail on a missing element and both fail on a disabled one. The danger comes from broadening the locator or negating the assertion.

One nuance separates a bad heal from a correct one that looks alarming. A heal that skips rather than weakens is right when the application is broken, and telling those apart takes human judgment, so read the Healer's reasoning before you treat a skip as a resolved failure. The broader question of where that line falls is covered in fixing Playwright tests with AI.

Both modes so far involve the agent getting something wrong. The third one appears when the agent is working exactly as designed.

Failure mode 3: UI-driven state setup

State arrives through clicking, because the browser is the agent's only instrument. Asked to test checkout with a saved card, it navigates to the login page, fills out the form, goes to account settings, adds a payment method, and then arrives at the product page. Every one of those steps worked when the agent ran it. That is why they are in the file.

In CI, those steps are the wrong approach. Login can hit a rate limit. The settings page can render an animation that outruns a timeout. The payment form can change in a sprint that has nothing to do with checkout. When any of that happens, the test fails on a screen it was never meant to cover, and whoever picks up the failure spends their time on a login bug that does not exist.

The fix moves the setup out of the test body. Authentication becomes storageState from a setup project, data creation becomes an API call in a fixture, and the test body starts on the page under test. One caveat: storageState captures cookies, localStorage, and IndexedDB, but not sessionStorage. Apps that store auth tokens in sessionStorage (common with OAuth PKCE flows) need addInitScript() to restore that state. The guide to testing authentication with Playwright covers the full range of patterns.

import { test as base, expect } from '@playwright/test';

type Fixtures = { savedCard: { id: string } };

export const test = base.extend<Fixtures>({
  savedCard: async ({ request }, use) => {
    const res = await request.post('/api/payment-methods', {
      data: { last4: '4242', brand: 'visa' },
    });
    const card = await res.json();
    try {
      await use(card);
    } finally {
      await request.delete(`/api/payment-methods/${card.id}`);
    }
  },
});

Two details in that fixture are worth reading carefully. await use(card) blocks for the entire test, so the code after it runs once the test and its hooks have finished with the fixture. The try/finally wrapping that call guarantees the delete runs whether the test passes or fails, because use() throws when the test body throws.

Treat that pattern as the default for any fixture that creates backend state, rather than an edge case. Setup goes through the API, deletion is guaranteed once the card exists, and the test body holds only the behavior you are checking.

The nuance here matters for how you talk to the agent. Interactive MCP sessions require a UI-driven setup because the agent has no other way to reach a logged-in state while exploring. Review is where you decide which of those steps belong in the test body and which belong in a fixture. The interactive loop itself is covered in Playwright MCP.

Knowing the three mechanisms turns review from an instinct into a procedure. What follows is the procedure.

The pre-merge review checklist

Run these seven checks in order on any pull request touching agent-generated specs. Each one catches something a conventional review does not look for.

Check 1: do the assertions verify business outcomes, not side effects?

Look for tests whose only assertions are toHaveURL, page title, or the visibility of a container that renders regardless of success. Add one assertion per scenario that names the outcome, whether that is a confirmation number, a total, or a status that changed. If the agent could have written the assertion without knowing your requirements, it did not verify a business outcome.

Check 2: are locators specific and role-based?

Look for .first(), .nth(0), CSS class selectors, and XPath with position predicates. Replace them with getByRole('button', { name: 'Add to cart' }), or add a data-testid where no accessible name exists. Generation source changes how much scrutiny this needs. Agents driving a real browser through MCP read the accessibility tree and generally produce good locators, so a clean pass is expected and a positional locator is worth a second look. The broader case for semantic locators is in designing tests that survive UI refactors.

Check 3: is state setup in fixtures, not in the test body?

As of 1.56, the Generator writes each spec on top of a seed file, tests/seed.spec.ts, which holds the shared imports and setup every generated test inherits. That seed is the right place for authentication and common fixtures, and a generated test that repeats login inside its own body is often carrying setup the seed should own. Read the seed alongside the spec before deciding a setup step belongs in the test.

Look for page.goto('/login') inside the test body, form fills against screens that are not under test, and navigation through settings or admin pages. Move authentication to storageState and data creation to API calls in a fixture. Tests that verify the login flow itself are the exception, since interface interaction is the behavior under test there. Everywhere else, a test that logs in through the interface will eventually fail for reasons unrelated to its purpose.

Check 4: are there any waitForTimeout calls?

Search for waitForTimeout. Each call is a fixed sleep, and the number encodes how long something took on the machine where the agent ran rather than on your CI runner.

Replace it wherever an observable condition exists, whether that is expect(locator).toBeVisible() for rendering or page.waitForResponse() for a network call. Keep it only where nothing observable marks the end of the wait: a debounce with no resulting request, a CSS animation with no transitionend event you can hook, or a third-party widget that initializes without emitting a ready signal. Agents insert these after watching a real delay, and the delay was real while the number was arbitrary.

Check 5: does each test clean up its own state?

Look for any write that outlives the test. That can be a POST through request, a PUT or PATCH against existing data, a record created by clicking through a form, or a direct database helper.

Each needs a matching cleanup path, which in a fixture means the code after await use(). Namespace created records with a unique suffix as well, so anything that does leak cannot collide with another test. Agents rarely write cleanup, because a teardown path is invisible from the browser.

Cleanup is also the main reason agent-generated tests break under --workers > 1. The agent wrote each test against a fresh browser on a quiet machine, so it never hit the shared-state collisions that parallel execution produces. If two tests create the same record without namespacing, one of them fails on the uniqueness constraint and the failure has nothing to do with the feature under test. The mechanics behind this are in sharding vs workers, and the broader category in Playwright anti-patterns.

Check 6: is the Healer's diff additive, not subtractive?

For any test the Healer touched, meaning any test the automated repair agent modified rather than one a person edited by hand, compare specificity before and after. Named locator to unnamed, getByRole to a raw selector, positive assertion to negated assertion, removed assertion: each is a signal that the heal made the test easier to pass rather than more accurate.

When the diff is subtractive, find out why the original check failed before accepting the replacement. A locator that broke because a button was renamed needs the new name. An assertion that broke because the feature broke needs a bug report.

Check 7: did the Healer skip anything?

Start by grepping the diff for test.fixme and test.skip. This takes seconds and catches the outcome nothing else does. Playwright reports both as skipped, and a run whose only non-passing tests are skipped exits zero, since the runner offers no option to fail on skips. A suite carrying silent skips is a green suite that has quietly stopped covering the flows those tests owned.

The two markers mean different things and both deserve scrutiny. test.skip marks a test as intentionally not applicable, often conditionally by browser or environment, and a conditional skip that predates the agent is fine. test.fixme marks a test as known broken and awaiting repair, which is the marker the Healer reaches for when it cannot make a test pass.

A skip is a defensible outcome of healing and never a merge-ready one. When either marker appears in an agent diff, the resolution is a tracked issue that names the broken behavior, then a merge.

Seven checks handle everything visible in a diff. The next question is what a perfect review still lets through.

What agents cannot catch even if you review correctly

Every check above reads one artifact. Review stops at that boundary. Three problems lie outside it: coverage drift from non-deterministic generation, agent-authored failures that mimic human flakiness, and healing that erodes strictness over time.

These are monitoring problems, not review failures. A reviewer reading a perfect diff has no way to see them, because the evidence sits in run history that does not exist while the pull request is open.

Non-deterministic generation producing coverage drift

Generation is not deterministic. The same plan run twice can produce tests with different assertion styles, variable names, or flow structures, since the model samples rather than compiles. Each variant is well-formed on its own, so each clears a checklist pass. Merge them across two sprints, and you end up with two tests for one behavior, each checking different things about it.

The risk is not flakiness. Two tests with conflicting assertions about the same feature would fail consistently, not intermittently. The risk is coverage drift: the suite grows without anyone knowing what is actually covered. One generated test checks the order total, another checks the confirmation heading, and neither checks whether the payment was actually charged. Both pass review, both pass CI, and the gap between them is invisible unless someone maps coverage to requirements rather than to test count.

Agent-authored failures that look like ordinary flakiness

Agent-authored failures show up as timeouts and locator errors, the same symptoms as a human-written flaky test. The difference is in the pattern, not the symptom. Human-authored flakiness clusters around a real timing dependency: it recurs on the same step and responds to a wait fix. Agent-authored failures cluster around when the test was generated, so a batch of tests written in one session fails in ways that share no application behavior.

That difference is invisible without knowing which tests came from an agent. A test that broke because the agent's context degraded mid-session looks exactly like a locator problem. That sends debugging down the wrong path, and the fix that gets applied does not hold. Tagging agent-generated tests at authoring time is what makes failure analysis scopeable by source, and it is why quarantine thresholds for agent-authored tests belong in a separate bucket.

The Healer creating a passing test suite that is less strict than the one before it

Check 6 asks a reviewer to judge the diff in front of them, which is all a single review can see. Each heal lowers the specificity bar slightly, every individual heal looks defensible on its own, and the suite drifts from its original intent over months without anyone having a reason to object. Catching that requires aggregating healed-test history across revisions rather than sharpening the per-diff check.

Three metrics make this drift visible. First, the number of healer-touched tests trending upward week over week. Second, the ratio of named getByRole locators to positional ones across healed tests. A ratio moving toward positional means the Healer is accumulating drift. Third, any test healed more than twice without its underlying failure being filed as a bug. That last one is the sharpest signal: either the application changed in a way the test needs to absorb, or the Healer keeps finding a way to make it pass.

Those are questions about run history, not about a file. Any cross-run reporting layer can answer them. Tracking suite health over time covers the approach, and combining AI and human QA covers where each layer owns responsibility.

Both layers need somewhere to live in your workflow, or neither one happens on a busy Friday.

Practical integration: building the review into your workflow

Two changes turn the checklist into something that runs without anyone remembering it. The first puts the seven checks into the pull request template so the review is enforced by the merge process. The second tags generated tests at authoring time, so failures can be attributed to their source later.

A PR template that enforces the review

Put seven checkboxes in the pull request template, one line each. Require them on any pull request that touches your generated test directory. Enforce it with a CODEOWNERS rule or a workflow that requires the boxes when changed paths match tests/generated/** or wherever your team routes agent output.

- [ ] Assertions verify business outcomes, not navigation alone
- [ ] Locators use role and accessible name
- [ ] Setup lives in fixtures or storageState
- [ ] No waitForTimeout
- [ ] Created data is cleaned up
- [ ] Heal diffs preserve specificity and still target the original element
- [ ] No test.fixme, and every test.skip has a stated condition

The last item is split deliberately. test.fixme in a generated diff means the Healer gave up on a test and left it silently uncovered, which is never merge-ready. A test.skip with a documented condition, such as a browser the feature does not support, is ordinary practice and stays.

The template handles the review. Attribution needs a second change, since a checklist tells you a test was reviewed and says nothing about who wrote it.

Tagging agent-generated tests at the source

Add the tag when the test is written, either through the Generator prompt or a post-processing pass before the pull request opens. Playwright gives you two mechanisms, and they do different jobs.

The tag option makes tests filterable on the command line.

test('checkout records an order number', { tag: '@agent-generated' }, async ({ page }) => {
  // ...
});

The annotation option carries structured metadata into the report.

test('checkout records an order number', {
  annotation: { type: 'agent-generated', description: 'Generator 1.56' },
}, async ({ page }) => {
  // ...
});

Both are fields on TestDetails, the optional second argument to test() in the Playwright API. Both compile under tsc --strict, and both reach the JSON report on 1.56 in different places. Tags appear on the spec, annotations on the test result.

The annotation field is singular because it accepts one entry or an array, and those entries land in the testInfo.annotations array at runtime. The declarative form also records the source location, which a runtime push does not.

The decision rule is short. Reach for tag when you want to filter runs on the command line. Reach for annotation when you want structured metadata in the report, such as which Generator version produced the test.

You can push onto testInfo.annotations inside the test body, and the array is a property rather than a method:

test('example', async ({ page }, testInfo) => {
  testInfo.annotations.push({ type: 'agent-generated', description: 'Generator 1.56' });
});

One caveat from the Playwright API reference. Changes made to testInfo.tags while a test is running are not visible to reporters, so tags have to be declared rather than added in the body. Annotations pushed at runtime do reach the report, which is why the push above works and a runtime tag would not.

The payoff is filtering. --grep @agent-generated scopes a run to agent-authored tests, which is what lets you review that population on its own rather than reading it averaged into the rest of the suite.

Wrapping up

Playwright Test Agents produce tests that compile, run, and pass. The gap between that and CI-ready comes down to three things the agent cannot know: what you wanted verified, whether its repair preserved the original intent, and which of its setup steps belong in a fixture. Seven checks close that distance before a merge.

What survives the review is a different kind of problem. Generation drift, agent failures that mimic flakiness, and healing that erodes specificity over months are patterns across runs, and no reading of a single file will surface them.

Review the artifact in front of you, and use cross-run history to catch the patterns a single file cannot show. Generated tests reach CI-ready with both layers in place, and with only one of them, you are shipping tests that pass for reasons nobody has checked.