Cypress vs Playwright for tech leaders: cost and risks analysis

Playwright has recently surpassed Cypress in NPM weekly downloads. Let's try to understand why Playwright became more popular and discuss:

• The cost of using a testing framework
• Why teams prefer Playwright over Cypress
• The path of migration from Cypress to Playwright

The cost of ownership

Real-world scenarios require a testing framework to do more that just basic execution of test scenarios - it has to accommodate other needs of modern development teams. For example:

• Debugging and troubleshooting, especially in CI, because when a test fails in CI it is hard to understand what went wrong when rerunning locally.
• Performance and cost-effectiveness, because waiting for tests completion wastes your organization's resources, requires more powerful CI machines and slows down the feedback loop.
• Reporting and analytics, because your testing suite will eventually grow and you will need to start measuring its performance like flakiness, duration, failures, etc.
• Compatibility with IDE and AI assistants, because it makes your developers more productive and helps them to write, debug and troubleshoot tests faster.
• Tests management, because eventually you will need to organize your tests by team, functionality, prevent duplication (hence, waste of resources).
• Developer experience, because engineers are the ones who write and maintain the tests, and if the framework is hard to use, slow, and flaky, they will waste their time on fixing the tests instead of writing new features.
• Full access to browser capabilities like cross-domain iframes, real browser events, testing multiple tabs or browsers, session storage - eventually, your application will require testing those scenarios.
• Generating artifacts like code coverage, visual diffs, accessibility reports, Lighthouse reports, because you will need to track the quality of your application beyond the functional tests.

Once you've picked the framework and invested into creating a comprehensive suite of tests, integrated it into your CI pipelines, created custom integrations and complimentary tools, it becomes a critical part of engineering infrastructure. Both Cypress and Playwright offer a solid list of out-of-the-box features, however, beware of the risks and hidden costs that are not apparent at the first glance:

• Vendor lock-in, switching testing frameworks is usually the last activity your organization wants to deal with, however your team can be blocked from accessing crucial features without having a commerical arrangement with the framework owner. Depending on the vendor and its commercial success (or lack of thereof), continuing using a testing framework can become too expensive.
• Performance and resources will affect the cost of running the tests in CI, flaky framework and CI reruns waste developer time on waiting for tests to complete.
• Highly specialized paradigms that are only applicable to a particular framework can lead to challenges with finding talent or adoption - it can be difficult to hire professionals who are experienced with the framework, or training your existing team to use it.
• Ecosystem and adoption, because the framework may not be maintained or supported in the future, or it may not have a large community that can help you with the issues you may encounter.

Let's try to summarize those aspects in the table below:

Feature	Cypress	Playwright
Parallelization	🔒 Vendor locked workaround cypress-split	Included: Test Sharding
Smart orchestration	🔒 Vendor Locked	Available with Orchestration for Playwright
Debugging	🔒 Vendor locked	Included: Screenshots, Videos, Traces ⚡️ with Currents Dashboard
Reporting & Analytics	🔒 Vendor locked Only high-level reports are available	3rd party providers available ⚡️ with Currents Dashboard
Accessibility Testing & Reports	🔒 Vendor locked	Included: Accessibility Testing
Flakiness analysis	🔒 Vendor locked	Included: Flakiness ⚡️ with Currents Dashboard
Code, CSS Coverage	Possible with a plugin	Included: Coverage
Visual regression testing	Possible with a 3rd party plugin	Included: Visual Comparison
Lighthouse reports	Possible with @cypress-audit/lighthouse	Possible with playwright-lighthouse
Component testing	Included Component Testing	Possible with Storybook Included in experimental mode
Tags	Run level only not supported for individual tests	Included: Tag Tests
Grep tests	Requires @cypress/grep	Included: Filter Tests
Cross-domain iframe	with chromeWebSecurity: false	Included: Frames
Multiple tabs / browsers	Not supported officially but may be possible in future with Puppeteer	Included: Browser Context
Session storage	Included cy.session	Included: Test Fixtures, Session Storage
Custom commands	Included Cypress.Commands.add()	Included: Test Fixtures, plain JS functions
BDD	Possible with cypress-cucumber-preprocessor	Possible with playwright-bdd
UI mode	Included Cypress App	Included: UI Mode
API testing	Included Cypress API Testing	Included: API Testing
Real browser events	Possible with cypress-real-events	Included: Native Events
Concurrency	One test at a time	High concurrency run multipe tests simultaneously with Workers
Performance	⚡️	⚡️⚡️⚡️
Resource usage	🏋️🏋️🏋️	🏋️
CI cost	💰💰💰	💰
Specialization	🥋🥋🥋 specialized syntax like chaining, custom commands, lacks async/await	🥋🥋 generic syntax, specialized paradigms like fixtures
Ecosystem	⭐️⭐️ artificially limited API, plugins ecosystem	⭐️⭐️ no plugins API, fully open reporter APIs
Adoption	⭐️⭐️⭐️	⭐️⭐️⭐️

* as of July 2024

Why teams prefer Playwright over Cypress

To put it simply, Playwright is more efficient - it is faster, more robust and cheaper.

Performance

Teams at Business Insider found that Playwright could test more scenarios (without third-party libraries) more reliably than Cypress (with third-party libraries), all while being faster and more flexible.

Ramp reported that the average test duration has been reduced by 32%. Additionally, Lingvano noted a 70% reduction in CI testing time.

Checkly compared the speed of execution on synthetic and real-world scenarios and found that Playwright is the fastest among the tested frameworks: Playwright tops the ranking for real-world scenarios with Cypress being on average ~23% slower compared to Playwright.

Speed

Indeed, the difference in performance between the frameworks is noticeable. For instance, a simple test that opens a page and asserts the title shows that Playwright is faster than Cypress (videos off) - 1.89s vs 6.59s:

// Cypress
it("opens a page and asserts the title", () => {
  cy.visit("https://example.com");
  cy.title().should("eq", "Example Domain");
});

$ /usr/bin/time -lh npx cypress run --spec ./cypress/integration/simple.spec.js \
  --browser chrome
	6.59s real		3.71s user		1.03s sys


// Playwright
it("opens a page and asserts the title", async ({ page }) => {
  await page.goto("https://example.com");
  expect(await page.title()).toBe("Example Domain");
});

$ /usr/bin/time -lh  npx playwright test ./simple.spec.ts
	1.89s real		1.02s user		0.42s sys

For a more comprehensive comparison, check out the Cypress vs Playwright speed comparison.

The difference can be attributed to the fact that Cypress is a (quite heavy) Electron application that needs to be downloaded from a CDN (~500Mb), while Playwright is a lightweight NodeJS package with only a few dependencies and a small footprint of just ~10Mb.

// Playwright (1.43.0)
$ du -sxh ./node_modules/playwright*
3.1M	./node_modules/playwright
7.4M	./node_modules/playwright-core

// Cypress (13.12.0)
$ du -sxh /Users/username/Library/Caches/Cypress/13.12.0/Cypress.app
539M	/Users/username/Library/Caches/Cypress/13.12.0/Cypress.app

The performance gaps have a direct impact on CI / infrastructure cost - longer startup times and slower execution mean you'll need to pay more or invest more engineering time in optimizing the CI by running tests in parallel (which is locked behind a paywall in Cypress). There are also indirect costs associated with developer productivity, e.g., longer feedback loops, more flakiness, and more time spent on debugging and troubleshooting.

Concurrency

Another aspect that affects the performance, especially in CI environments, is the concurrency model those frameworks employ:

• Cypress can split tests across multiple machines, and each machine will run exactly one test at a time
• Playwright can split the tests across multiple machines, and each machine can run multiple tests simultaneously

The highly concurrent model means faster execution and more efficient use of resources, which translates into faster CI feedback and lower costs.

Cypress parallelization is a paid feature, although, with certain limitations, you can use 3rd party solutions like cypress-split, Currents or Sorry Cypress.

Reduced Flakiness

We have analysed hundreds of real-world projects implemented both in Cypress and Playwright. This data comes from applications of various sizes and complexities, as well as different CI providers and setups.

It is possible to eliminate flaky tests completely with both Cypress and Playwright. However, according to our analysis and customer feedback, Playwright teams have a lower chance of experiencing flakiness.

• On average, teams are experiencing 15.60% less flakiness with Playwright (0.72%) compared to Cypress (0.83%) - the absolute difference is 0.11% in favor of Playwright, the relative difference is 15.60% in favor of Playwright;
• Cypress not only has a higher average flakiness rate but also a wider spread - i.e., Cypress projects experienced more variability in flakiness;
• Cypress data had a more extended tail towards higher flakiness rates, indicating more extreme values than Playwright (even after removing the outliers).

See Flakiness Analysis of 1000s of Real-world Cypress and Playwright Projects for more details.

Versatility

During the last couple of years Cypress established itself as a solid solution for browser-based testing, although with a few "quirks".

• its architecture puts limitations on what can be tested - e.g. multi-tab testing is not possible, also cypress simulates browser events instead of using native ones (both limitations need experimental or 3rd party plugins to overcome)
• an opinionated approach to authoring and executing tests using chained commands forces engineers to learn and use specialized syntax
• certain features, arguably even essential ones like running tests in parallel or getting detailed tests execution reports, are locked behind a paywall
• coverage, accessibility and analytics are locked behind a paywall or require 3rd party plugins

Meanwhile, Playwright team managed to use browsers' modern APIs to create a lightweight solution that doesn't have those limitation. Many of Cypress' features that are either paid or require a 3rd party plugin are integral part of Playwright and can be used out-of-the box.

The migration path

Many teams are intrigued by the effectiveness of Playwright, but their testing suite is already implemented in Cypress. We have collaborated with dozens of organizations that, due to various reasons, opted to do the migration.

Typically, the migration plan from Cypress to Playwright typically consists of 4 stages:

1. Prepare the testing infrastructure
2. Write new tests in Playwright, migrate high-value tests
3. Fine-tune the CI setup and reporting
4. Migrate the existing tests

The consensus is that you don't have to migrate all the tests at once but rather gradually. New tests are written in Playwright, while the existing tests are migrated based on the available resources.

We've seen that after the initial spike of preparing the infrastructure, teams were able to switch to low-intensity mode when all the new tests are written in Playwright, and high-value and important tests are being gradually rewritten within 2-3 months.

The cost of migration depends on the size of the testing suite, the complexity of the application, the team's experience and the available resources.

See Cypress to Playwright Migration Guide for a more detailed analysis.

Summary

Playwright became more popular because it is more efficient - faster, more robust and cheaper. Teams can test more scenarios with less flakiness and run CI tests faster using fewer resources.

• Cost of Ownership: Playwright offers more features out-of-the-box, reducing the need for third-party plugins and "walled garden". The cost of running tests in CI is lower due to the faster execution and more efficient use of resources.
• Performance: Teams at Business Insider and Ramp report up to 20x improvements in performance. Additionally, other publications note a 70% reduction in CI testing time.
• Reduced Flakiness: an analysis of 100s of real-world projects found that teams experience less flakiness with Playwright.
• Robustness: according to Insider team, Playwright allows testing more reliably and faster scenarios without needing third-party libraries.
• Efficient Use of Resources: Ramp discovered that Playwright's architecture is more scalable, performant, and less resource-consuming. These aspects allowed them to reduce their CI machines fleet by 62%.
• Migration Overview: the migration from Cypress to Playwright can be done effectively. Teams were able to gradually migrate the tests after the initial 2-3-week spike of preparing the infrastructure, then switching to a low-intensity effort of gradually migrating the tests. The key to success is to keep improving the testing infrastructure and make it easy to write new tests.

How Currents can help?

Our team at Currents has been working on providing the best-in-class cloud-based reporter for Playwright. Currents is not just a reporter - it is a platform that integrates with your testing suite and workflows, improving your testing suite's performance, reliability and efficiency.

• We help reduce CI execution and cost via improved orchestration for Playwright, unlocking the usage of cloud spot instances for E2E tests
• We provide a near-realtime streaming of step-level test results, together with screenshots, videos and traces for better debugging and troubleshooting
• We help improve the quality of your tests by proactively identifying the flaky tests, analyzing trends and providing actionable insights
• We integrate with your existing tools and workflows, like Slack, GitHub, GitLab, etc.

For companies migrating from Cypress to Playwright - you can run both Cypress (up to version 12.17.4) and Playwright, using Currents. We offer dedicated plans adjusted to facilitate the migration. Please reach out for more information.

Join the growing community of teams using Currents to run their Cypress and Playwright tests

Trademarks and logos mentioned in this text belong to their respective owners.