Playwright Reporter for Currents v1.0.0 ✨

Version 1.0.0 is a big deal!

This release concludes a year of collaboration with the Playwright community, learning from our customers, Playwright ambassadors, and industry experts. We have worked hard to create the best-in-class Playwright reporter and to tackle the most challenging issues in the Playwright ecosystem.

After a few weeks of testing and iterative improvements, we are ready to announce that version 1.0.0 is out of beta.

Now, let's dive in and explore what's new in this release.

No Breaking Changes

There are no breaking changes - you can use the same API and configuration (including pwc and configuraion).

Fully Parallel Mode

Playwright supports multiple levels of concurrency:

• a test suite can be split between different machines using sharding
• each individual machine can have multiple workers (according to CPU performance or explicit configuration)

The workers will run the tests in parallel, allowing for a speedup in execution and full utilization of a machine's resources.

However, by default, Playwright runs the tests of the same spec file serially — one after another. So, even if you have a powerful machine capable of supporting multiple workers, it can be underutilized. For example, if your machine has 4 workers and sharding assigns only one spec file, the tests will be executed one by one, and only one worker will be utilized at a time.

Enabling fullyParallel: true (see configuraion details) allows running tests from the same spec file in parallel on different workers. This way, all available workers can be utilized, and the tests will be executed faster.

Note, that it requires the tests to be isolated and not to share any state - Playwright will run the tests on different workers or even on different machines (a.k.a shards).

To enable running individual tests from the same spec file on different workers, set fullyParallel: true in your configuration file.

Version 1.0.0+ of the reporter is compatible with fully parallel mode and will collect the results of concurrent parallel executions from distributed shards and workers. Moreover, the reporter will display the results of each individual step, which is crucial for debugging failed CI executions.

Step-level Reporting

Traditionally, Playwright reporters generate results after the run completion, when all the results are ready. This approach works well when executions are successful. However, when an execution fails to complete, no results are reported at all.

For example, imagine running a spec file with 10 tests on a CI machine. Nine tests can be completed successfully, but the last test can cause a crash. If we report after completion, no results will be reported at all, even for the nine finished tests.

This problem affects not only the test results per se but also the generating and uploading of artifacts like traces, screenshots, and videos. We observed this issue occurring in different scenarios, ranging from infrastructure issues to Playwright bugs.

We wanted to provide a crash-resilient solution that would help debug failed CI tests and provide as much information as possible as soon as it is available. The Currents reporter streams the results for each individual step, including step-level lifecycle events (e.g., step started, step ended) as soon as they're available. So, even if an execution crashes, Currents will capture the last step that caused the failure.

In addition Currents uploads the generated artifacts to a cloud storage as soon as they are available, so you can access them even if the execution crashes.

In addition to being resilient to crashes, collecting data about individual steps allows Currents to present detailed information via its web interface. The results are stored in cloud storage, providing easy access to the entire team during Playwright tests execution. Think of it as a light version of Playwright traces that are available immediately and are not dependent on the test completion or the successful upload of a trace file.

Having access to such detailed information unlocks more insights, like step-level analytics and performance analysis.

It's important to note that step-level reporting is available because the Playwright team and other contributors have designed and provided the necessary APIs to capture the lifecycle events of the test execution.

We are grateful for their excellent work and the opportunity to build on top of it 🙏.

Orchestration for Playwright

Orchestration for Playwright allows to significantly decrease the duration of parallel CI Playwright runs. A few of our customers saw up to 40% increase in CI runs speed, compared to the traditional Playwright sharding.

Traditional Playwright Sharding

The traditional Playwright sharding method, which splits spec files between available shards (machines) based on the filename, often leads to unbalanced workloads. Since spec files can vary significantly in execution time, this imbalance means that some shards may end up processing only long-running spec files, while others quickly finish with shorter tests, leading to idle resources.

As the testing suite expands, periodically changing spec file names to redistribute the load more evenly becomes a necessity for optimizing resource use and minimizing total execution time. However, this approach is manual and prone to errors.

Take, for example, a testing suite consisting of 4 spec files with varying durations:

• spec01: 10 minutes
• spec02: 10 minutes
• spec03: 3 minutes
• spec04: 2 minutes

With 2 machines, traditional sharding might distribute the files as follows, leading to an inefficient total execution time of 20 minutes due to one shard being heavily loaded while the other finishes quickly:

• Shard 1: spec01, spec02 (20 minutes total)
• Shard 2: spec03, spec04 (5 minutes total)

An optimal assignment strategy that considers the duration of spec files can significantly improve the efficiency of test execution. By balancing the workload across shards based on test durations, you minimize overall execution time and make better use of resources. For the example provided:

• Shard 1: spec01 (10 minutes) and spec03 (3 minutes), totaling 13 minutes.
• Shard 2: spec02 (10 minutes) and spec04 (2 minutes), also totaling 12 minutes.

The difference in execution time 13 vs 20 minutes, which is a 35% improvement.

Optimal Tests Balancing with Playwright Orchestration

Playwright Orchestration by Currents automates the process of achieving a balanced distribution of spec files based on their expected duration. Here are the key benefits:

• ✅ No need to manually update CI configuration when changing the number of shards - Currents automatically assigns the spec files to all available CI machines.
• ✅ No need to change spec files' names to achieve a better balance between the shards.
• ✅ No need to manually track spec files duration and adjust the sharding configuration.
• ✅ No complex configuration or setup - simply use the pwc-p command (included in @currents/playwright) to run your tests.

This approach has enabled us to reduce the execution time of CI Playwright tests by up to 40% for several of our customers, with minimal changes to their existing CI configurations.

Various Orchestration Strategies

The assignment of spec file can account for other factors like:

• expected failure
• flakiness rate

For example, Currents can prioritize the spec files that are more likely to fail. Combined with the Fail Fast Strategy, Currents will halt the execution across all the shards if a certian number of failures was detected.

This strategy allows teams to save on CI resources and receive faster feedback from their CI testing suite, making the process not only more efficient in terms of execution time but also more cost-effective and responsive to issues as they arise.

🧪 Experimental: Redriving Tests

An additional benefit of using Currents for balancing tests is the ability to automatically redrive tests from one machine to another.

For example - many cloud providers have an option to use Spot Instances for running workloads. Using Spot Instances can cost up to 90% lower, compared to the traditionally allocated resources.

However, spot instances can be terminated at any time, which can cause the loss of the test results. Currents can automatically redrive the tests from to-be-terminated instance to another machine. This way, the execution can continue without manual intervention.

We are currently testing this feature with a few of our customers, please contact us if you are interested in participating in the beta.

Improved Reporter Console Output

We have refactored the look and the feel of the reporter's console output for more precise and accurate presentation of the execution workflow.

• each attempt have its start and end timestamps, status details, detected errors, stdout and stderr output

• clear separation between tests flakiness and outcome status - determining a test's "status" and flakiness requires considering the outcome of all the attempts, the expected status and the execution mode (e.g. "serial" mode can generate extra attempts).

• inline links to individual test results is part of console output

Summary

Go ahead and install the latest version of the Playwright reporter: npm i @currents/playwright@latest.

We are excited to see how it will help you to improve your Playwright testing suite. Don't hesitate to reach out to us if you have any questions or feedback.

---

All third-party trademarks and materials (including logos, icons, marks and labels) referenced herein are the property of their respective owners.