Cypress parallelization tools comparison

This blog post is a courtesy of Laerte Neto. The original publication can be found here.

---

Cypress Dashboard

Cypress Dashboard is the official tool offered and supported by the Cypress company itself. It brings a complete solution to setup parallelization with automatic load balance and it also brings a very interesting dashboard to follow test executions.

Cypress dashboard main page: Test and debug faster with the Cypress Dashboard

Steps to implement Cypress Dashboard in any CI provider: Parallelization | Cypress Documentation

Price and features: Pricing

With Cypress dashboard, we basically have:

• Parallelization with Load Balancing controlled by AI
• Debugging and analytics tools
• Video recording and screenshots of everything as much as you want
• Flake detection system
• Jira Integration
• Premium support on Cypress bugs or new features

It is used by big companies like Disney and PayPal. More features are being added in the Cypress Dashboard currently, so if you go over there you will see some new features with a tag “coming soon”.

Sorry Cypress / Currents

Sorry Cypress is probably the best known workaround solution/tool for Cypress Dashboard.

It brings the parallelization and flaky detection system with a way more simple dashboard to follow test executions.

Currents is the Sorry Cypress’ solution in cloud that provides you the tool without any setup.

Let’s resume some points by comparing it with Cypress Dashboard:

Pros:

• It is free and unlimited by using your own CI/CD infrastructure and it is supported in many cloud platforms, such as AWS, Azure, and Google Cloud.
• It is not hard to set it up, since they provide us a good documentation, including a docker-compose file that you can build it in your server and have everything up and working, including database to keep the data persistent and track test execution history.
• We keep all the data in our side without sending any data to a third party server. It may be a advantage regarding security.
• The dashboard is simple but it offers the most used tools for test executions such as screenshots, video recording, error stack trace, and flaky detection.
• Currents is the Sorry Cypress’ cloud based solution, which is a way cheaper than the official Cypress Dashboard solution, and also does not bring any kind of server/cloud support and setup for your part. This solution is in beta stage (Update: It came out of beta in November 08th, 2021), but looked pretty stable and well functional during my personal tests.
• Currents declares to offer an Optimal Sorting feature to smartly distribute tests to reduce overall run time. I sent an email asking more details about it and the answer was: “We track the duration for each file and when orchestrating a new run, we sort the files to be tests by their expected duration and serve longest files first. The average duration is the moving average of last 2 weeks history.“
• Take a look in here to see a demo of Sorry Cypress or test the Currents through the 14 days free trial.

Cons:

• You are required to setup it up in your own infrastructure in case you want to use it without any cost, which means you may need to have a server or something like that separated for the Sorry Cypress server, and we may need to maintain it as well.
• During some tests in my project, it was observed an incompatibility within our Cypress Snapshot Comparison plugin (cypress-plugin-snapshots, version 1.4.4). So, we would need to remove it or look for another snapshots comparison solutions/plugins if we decide to move forward with Sorry Cypress.
• Cypress version upgrade can take up to 2 weeks on Currents.
• The information regarding the minimum/recommended hardware requirement to run it in our own cloud solution was not found.

Cypress-Parallel plugin

The tool called cypress-parallel declares itself to reduce up to 40% your Cypress suite execution time parallelizing the test run on the same machine. It does not offer any dashboard or management solution, and it only runs test in parallel by using threads.

Pros:

• Simple to setup and run tests in parallel.

Cons:

• Pretty new solution, basically without any support besides the community (which is very small yet). However, the developer seems to interact a lot with the community in the GitHub page.
• It runs tests in parallel by using the same job instance, which makes it challenging to follow test execution or while debugging in the logs.
• It was observed an issue while running it in a Bitbucket pipeline, which failed the job even though the test was successful. However, it may be a problem with my own cloud infrastructure, so further investigation would be necessary to find out an answer for that.
• It was also observed an incompatibility within our Cypress Snapshot Comparison plugin (cypress-plugin-snapshots, version 1.4.4).
• At the moment, it just runs tests in parallel, without any tool or dashboard to support executions.
• There is no AI feature to load balance, so the tests are chosen in the order that your spec files are disposed in your project folders.

Manual CI Approach

This approach is basically to use your own CI provider infrastructure to run your tests in parallel and use the mockawesome reporter as the dashboard or another dashboard provider like Grafana. In this approach, we would basically spilt the tests by spec files on “n” machines by using the “spec” flag.

Pros:

• You can use your own infrastructure without any plugin, side-server, or tool.
• No need to maintain any side-server.
• It is a pretty simple solution, without needing any setup but only your usual pipeline setup.
• The mockawesome reporter can be easily used as a simple dashboard for each execution.

Cons:

• You will need to manually split and maintain the specs files scattered/organized over the jobs/machines.
• The load balance is totally manual by the splitting approach mentioned before.
• The test execution can be followed basically by logs. Grafana or some other dashboard can also be used, but you would be required to set them up entirely.
• In case of choosing a static reporter, the mockawesome reporter, for example, is generated only at the end of the execution, so there is no way to follow the execution while it is active.
• As this is a manual approach, you would not be able to guarantee that it would be the fastest execution in parallel (probably it will not). However, depending of the CI provider, it can still be really fast and be worth.

Conclusions

Cypress Dashboard is by far the most complete solution. However, it is expensive if compared with Sorry Cypress (free), which also brings some of the most relevant features that Cypress Dashboard has to guarantee a good parallelization approach and test management. Cypress-parallel is pretty new in the market, so at this point we would not consider it.

Besides the manual effort, the Manual CI Approach can also be something we can think about it as a backup solution (without any cost) if you decide to move on with Cypress Dashboard or Sorry Cypress/Currents. In this case, you may not even need to use it any time soon, so it may be a good free backup solution in case your main tool/solution is down for some reason.

Currents, from Sorry Cypress, can be something you could also keep in mind, because it brings Sorry Cypress in cloud, so you would not need to be worried about maintenance and updates. It was very fast and stable during my tests, which makes me very confident about it.

Therefore, if you want the official and most complete solution with total support by Cypress, and that it is already used by big companies, and money is not the problem, go on with Cypress Dashboard. However, if you are looking for something that is also good, and without lots of analytics, but that does the job and offers a pretty decent dashboard as well, go with Sorry Cypress (using your own infrastructure) or Currents (for a cloud solution without any setup). Finally, there is always the Manual CI Approach without any financial cost, that may be also interesting to use it as a backup solution that maybe will not even be used on a daily bases.

Thanks for reading!