Playwright 1.64: Shuffle, Test Locks and Breaking Changes

A suite that has run in the same order for a year can hide tests that pass only because the test before them left the right state behind. Playwright 1.64 adds a --shuffle option that schedules tests in a random order to find them. It also adds a lock option to test.describe.configure(), which adds test locks to all tests in a file or a group. Tests that share a lock name never run at the same time, while all other tests continue to run in parallel.

The same release changes behavior your CI depends on. Elements inside hidden iframes now count as hidden. Device descriptors now forward the screen size, so window.screen and media queries see the emulated screen. And --update-snapshots=missing now passes the run when tests only create missing snapshots.

In 1.64, tests that only create missing snapshots pass under --update-snapshots=missing, and elements inside an iframe that is not visible are now considered hidden by actions, locator.isVisible() and expect(locator).toBeVisible(). The fix is an order of operations: clear the breaking changes first, then turn on the new runner features.

What's new in Playwright 1.64

• In a breaking change, elements inside an iframe that is not visible, for example with visibility: hidden, are now considered hidden by actions, locator.isVisible() and expect(locator).toBeVisible().
• In a breaking change, the screen property is now forwarded from the device descriptors, so if you use ...devices['Desktop Chrome'] and alike, window.screen and media queries now see the emulated screen size.
• In breaking changes, --update-snapshots=missing now passes the test run, and JSX in your test files now follows your tsconfig.json and defaults to the automatic runtime from react/jsx-runtime.
• A new video style option takes CSS declarations for the point marker, the target highlight and the action title, and replaces the fontSize option, which is now deprecated.
• Videos are encoded with VP9 instead of VP8, and the cursor stays visible at the last action point.
• The test runner gets a new --shuffle command line option that schedules tests in a random order, and a new lock option in test.describe.configure() adds test locks to all tests in a file or a group.
• A new testProject.default option keeps a project in the config without running it by default, and a new type option of toHaveScreenshot in testConfig.expect stores all unnamed screenshots as WebP.
• New locator.within() combines two locators that you already have, reading in the natural order ("this button, within that dialog"). Relative locators such as locator.nth() or locator.first() are resolved inside each parent separately, which makes column-like queries straightforward.
• New page.getByRef() locates an element by its aria ref reported by page.ariaSnapshot() in the 'ai' mode.
• New includeShadow option in page.content() and frame.content() serializes open shadow roots as declarative shadow DOM.
• New signCount option sets the initial signature counter of a virtual credential.
• New fullConfig.filteredProjects lists the projects selected after applying the --project filter.
• New apiRequestContext.addCookies(), apiRequestContext.cookies() and apiRequestContext.clearCookies() manage cookies of a request context.
• New page.webmcp and frame.webmcp give access to the tools that a page registers through the experimental WebMCP browser API.
• Playwright MCP also supports WebMCP by default, with page-defined tools offered to the agent as webmcp_<tool>; pass --no-webmcp to opt out.
• playwright-cli exposes these WebMCP tools as well.
• Browser versions are Chromium 156.0.8078.4, Mozilla Firefox 157.0 and WebKit 27.2.
• The full list of changes is in the Playwright 1.64 release notes.

Run --shuffle nightly and keep the seed

Order-dependent tests stay green until something changes the order. A test that leaks state through a module-level variable, or depends on another test's side effects, passes when tests run in order and breaks when they run in a different one. --shuffle changes the order on purpose, so the shared state that breaks growing suites shows up as a failure instead of hiding.

What actually gets shuffled

With --shuffle, Playwright runs test files in a random order. Individual tests are shuffled too, but only in parallel mode, and tests that must run together, such as a serial suite, keep their order. Playwright runs a single file's tests in declaration order unless you parallelize that file. So in a file without parallel mode, a dependency between two of its tests stays hidden, however many nightly runs you shuffle.

The anti-pattern: shuffle on the PR gate

Adding --shuffle to the blocking PR pipeline without recording the seed is an anti-pattern. A random-order failure there blocks a pull request that never touched the tests involved. Playwright prints the seed at the start of the run, so the author can replay the order that failed from the PR job's log only while that log is kept. Once the log is gone, we'd expect a retry in a new order to pass and the dependency to stay in the suite as one more flaky test nobody fixes.

Run it nightly and keep the seed

Put shuffle on a nightly job instead, where a failure gets triaged rather than blocking a merge. Passing the seed back reproduces the same order:

# Nightly job: random order, seed printed at the start of the run
npx playwright test --shuffle
# Triage: replay the nightly order with the seed from its log
npx playwright test --shuffle 81723

Keep the seed with the nightly log, next to the failure, so whoever triages it can replay that exact order locally. A shuffled failure without its seed is a red run nobody can reproduce.

Filters are where replays go wrong. In a Playwright issue, a user reports that passing a test file right after the shuffle flag ran the whole suite with the file path as the seed and silently dropped the filter. When you combine the flag with a filter, check that the printed seed is the one you passed.

When a nightly failure lands, Currents Test History shows that test's results run by run across branches, so a test that has failed only since shuffle began stands apart from one that has always been flaky. History matches on spec file, project and title, so renaming the test detaches it.

Keep tests that share an account apart with test.describe.configure({ lock })

Some tests can't overlap because they share something you can't copy, like a live third-party account. In a Playwright issue, a user asked to declare a lock for the enclosing suite, because tests that share a live third-party account must not run concurrently.

The blunt fix is an anti-pattern: making every test that touches the account serial, or making the whole suite serial. That slows every test down to protect a few. Playwright's parallelism docs say using serial is not recommended and suggest making tests isolated so they can run independently.

How the lock option scopes

The lock option applies a named lock to every test in the enclosing file or group, nested groups included. The option adds to locks declared in test.describe() details and in earlier test.describe.configure() calls rather than replacing them. Tests that share a lock name never run at the same time, while every other test keeps running in parallel. Locks work across files, worker processes and projects, so two spec files that declare the same name stay apart:

// tests/billing/invoices.spec.ts
import { test, expect } from '@playwright/test';

// Shares one live payment-provider account with refunds.spec.ts
test.describe.configure({ lock: 'payments-account' });

test('creates an invoice', async ({ page }) => {
  // assumes use.baseURL in playwright.config.ts
  await page.goto('/billing/invoices/new');
  await page.getByRole('button', { name: 'Create invoice' }).click();
  await expect(page.getByText('Invoice created')).toBeVisible();
});

// tests/billing/refunds.spec.ts
import { test, expect } from '@playwright/test';

test.describe.configure({ lock: 'payments-account' });

test('refunds a paid invoice', async ({ page }) => {
  // assumes use.baseURL in playwright.config.ts
  await page.goto('/billing/invoices/paid');
  await page.getByRole('button', { name: 'Refund' }).click();
  await expect(page.getByText('Refund issued')).toBeVisible();
});

A test can declare several locks and runs only when all of them are free. Playwright acquires a test's locks before the test starts and releases them when it finishes. Repeated test.describe.configure() calls add locks, so a group that touches two shared resources declares both:

// tests/billing/export.spec.ts
import { test, expect } from '@playwright/test';

test.describe('invoice export', () => {
  // Repeated calls add locks: these tests wait until both are free
  test.describe.configure({ lock: 'payments-account' });
  test.describe.configure({ lock: 'shared-inbox' });

  test('emails the export', async ({ page }) => {
    // assumes use.baseURL in playwright.config.ts
    await page.goto('/billing/export');
    await page.getByRole('button', { name: 'Email export' }).click();
    await expect(page.getByText('Export sent')).toBeVisible();
  });
});

The cost of a lock outside parallel mode

In the default and serial modes, a locked file blocks every other test that shares its lock until the file's last test finishes. All tests in such a file run together in order, so a lock declared on any one of them is held for the duration of the whole file. Keep locked files small, and keep unrelated tests out of them.

Locks work around shared state; they don't remove it. Where you can give each worker its own account, do that, and save locks for the resources you truly cannot duplicate.

Write new baselines and verify old ones in one run with --update-snapshots=missing

A red run used to be how --update-snapshots=missing told you a baseline was new. In 1.64 that run is green.

Tests that only create missing snapshots now pass in the 'missing' mode of testConfig.updateSnapshots, so one CI run can write new baselines and verify the existing ones. When the option is not set, Playwright uses the mode now called 'default', which behaves as before: it writes missing snapshots and fails the test.

Treating a green run under --update-snapshots=missing as proof that no baseline was missing is the anti-pattern to drop on upgrade day. The run now passes after writing new baselines, so a pipeline that relied on red to flag them lets new baselines through without a signal. Add a step that fails the job, or commits the files, when the run wrote new snapshots:

npx playwright test --update-snapshots=missing
# The run passes after writing new baselines, so look for them explicitly
NEW_BASELINES="$(git status --porcelain --untracked-files=all -- '*.png' '*.webp')"
if [ -n "$NEW_BASELINES" ]; then echo "$NEW_BASELINES"; exit 1; fi

The modes are easy to mix up. In a Playwright issue, a user reported that the changed mode of the update-snapshots flag was adding missing screenshots, which they didn't expect. The docs say the 'changed' mode also creates missing snapshots. Set updateSnapshots explicitly for each CI job instead of relying on what happens when it is unset:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  // CI writes missing baselines and verifies the rest; local runs fail on a missing one
  updateSnapshots: process.env.CI ? 'missing' : 'default',
  expect: {
    toHaveScreenshot: { type: 'webp' },
  },
});

The type option of toHaveScreenshot in testConfig.expect stores all unnamed screenshots as WebP. We'd make that switch in a pull request of its own, so the format change and a real visual diff never land together.

Don't let default: false projects drop out of CI

testProject.default keeps a project in the config without running it by default, and it can make CI coverage disappear without a single red test.

To run a project with default: false, select it with the --project command line option. Locally, leaving such a project out is the point: configure every browser you care about and run only your main one.

import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  globalSetup: './global-setup.ts',
  projects: [
    { name: 'chromium', use: { ...devices['Desktop Chrome'] } },
    // Configured, but skipped by a plain npx playwright test
    { name: 'firefox', use: { ...devices['Desktop Firefox'] }, default: false },
    { name: 'webkit', use: { ...devices['Desktop Safari'] }, default: false },
  ],
});

The anti-pattern is adding default: false to save time locally, while the CI job still runs a plain npx playwright test. Firefox and WebKit stop running in CI, and no test fails to tell you, because the tests that would fail never start. In CI, select every project with --project:

npx playwright test --project=chromium --project=firefox --project=webkit

Then make the job check itself. fullConfig.filteredProjects lists the projects selected after the --project filter, and you can read it in global setup and in reporters. Compare it with the set CI should run, and throw when one is missing:

// global-setup.ts
import type { FullConfig } from '@playwright/test';

const CI_PROJECTS = ['chromium', 'firefox', 'webkit'];

export default async function globalSetup(config: FullConfig) {
  if (!process.env.CI) return;
  const selected = config.filteredProjects.map((project) => project.name);
  const missing = CI_PROJECTS.filter((name) => !selected.includes(name));
  if (missing.length > 0) {
    throw new Error(`CI skipped projects: ${missing.join(', ')}. Pass --project for each one.`);
  }
}

A job that drifts back to a plain run now fails loudly instead of quietly testing one browser.

Breaking changes that can turn CI red after the upgrade

In 1.64, the new style video option replaces fontSize, which is now deprecated.

Screenshots that shift with the emulated screen

If you use ...devices['Desktop Chrome'] or a similar descriptor, window.screen and media queries now see the emulated screen size. Device descriptors now forward the screen property, and a layout driven by a screen-size media query can render differently.

Regenerating every failing visual baseline after the upgrade is the anti-pattern here. Some of those diffs may come from screen forwarding rather than a regression, and updating them all at once buries the difference. Opt out first by setting screen to undefined, then review the diffs in a separate change:

import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  projects: [
    {
      name: 'chromium',
      // Opt out of screen forwarding until the baselines are reviewed
      use: { ...devices['Desktop Chrome'], screen: undefined },
    },
  ],
});

Remove the override once the baselines reflect the emulated screen.

Elements inside hidden iframes

Elements inside an iframe that is not visible, for example one with visibility: hidden, now count as hidden for actions, locator.isVisible() and expect(locator).toBeVisible(). In a Playwright issue, a user reported that a frame with visibility: hidden is incorrectly treated as visible. On 1.64, an assertion that such an element is visible fails, and an action on it targets an element Playwright considers hidden. Find the frames your app hides with CSS and fix the tests that reach into them before the upgrade lands.

JSX in test files

Playwright now compiles JSX in test files according to the jsx, jsxFactory, jsxFragmentFactory and jsxImportSource options in tsconfig.json, and defaults to the automatic runtime from react/jsx-runtime. If your tests render JSX, check those options before you bump the version.

The deprecated fontSize video option

The new style option takes CSS declarations for the point marker, the target highlight and the action title. Move your video config to style as part of the upgrade, not after.

Across a large suite, Currents Test Explorer shows each test's % Failure Trend against the previous period and which tests started to flake recently, so you see what the upgrade broke across the suite instead of one red run at a time.

What to do before you bump the version

Upgrade with shuffle off. The breaking changes bring their own failures, and fixing them first keeps them from mixing with order-dependent ones.

Two of the release's headline items don't belong in this plan. The new page.webmcp and frame.webmcp give access to the tools a page registers through the experimental WebMCP browser API, and the new locator.within() combines two locators you already have. Both are new APIs, so no existing test calls them. Leave them out of the upgrade branch. The release notes have examples for both when you want to try them.

Before you merge, run the upgrade branch once with CI's exact commands and project selection. Compare which tests and projects ran against the last 1.63 run. If fewer ran, find out why before you look at a single failure.

Once the upgrade branch is green, add the nightly --shuffle job with its seed in the log. --shuffle schedules tests in a random order, which helps to find tests that accidentally depend on each other. Add locks only where tests share something you cannot duplicate.