A Look at Playwright Parallelism
Playwright is an open-source automation library developed by Microsoft, designed for testing web applications across multiple browsers. It enables developers and testers to write scripts that simulate real user actions when interacting with web pages. Playwright supports major browsers like Chrome, Firefox, and Safari, including their headless modes. Recently, it has gained a lot of popularity for its robustness, ease of use, and cross-browser compatibility. In this blog post, we will take a look at one very useful aspect of Playwright: running tests in parallel.
Parallelism in Playwright
Parallelism in Playwright refers to running multiple test spec files or even test cases within a spec file simultaneously, greatly improving test execution speed. This is achieved by running the tests in worker processes, where each worker process is an OS process, running independently, orchestrated by the test runner. All workers have identical environments, and each starts its own browser instance.
Parallel execution is particularly beneficial in continuous integration and continuous deployment (CI/CD) pipelines, reducing overall build and testing times. Playwright's architecture inherently supports parallelism, and most modern test runners integrating with Playwright can leverage this feature. The parallel execution feature makes Playwright a highly efficient tool for large-scale web application testing.
Enabling Parallelism
By default, if you scaffold the project using npm init playwright@latest, parallelism is already enabled. Assuming that Playwright's configuration includes three browsers and there is a single spec file with two test cases, the total number of tests that Playwright needs to execute is 3x2 = 6.
import { test, expect } from '@playwright/test';
test('has title', async ({ page, browserName }) => {
console.log(`Running 'has title' test in worker ${process.env.TEST_WORKER_INDEX} using browser ${browserName}`)
await page.goto('https://playwright.dev/');
// Expect a title "to contain" a substring.
await expect(page).toHaveTitle(/Playwright/);
});
test('get started link', async ({ page, browserName }) => {
console.log(`Running 'get started link' test in worker ${process.env.TEST_WORKER_INDEX} using browser ${browserName}`)
await page.goto('https://playwright.dev/');
// Click the get started link.
await page.getByRole('link', { name: 'Get started' }).click();
// Expects page to have a heading with the name of Installation.
await expect(page.getByRole('heading', { name: 'Installation' })).toBeVisible();
});
Playwright will decide on how many workers to use based on several factors:
- Hardware support: Depending on the number of CPU cores and other system resources, the operating system will decide how many processes Playwright can spin up.
- The
workersproperty in theplaywright.config.tsfile. - The
--workersargument to theplaywright testcommand. For example,npx playwright test --workers 4.
The --workers argument overrides the workers property in the configuration. However, in both cases, the number of workers can go up to the number of processes that Playwright can spin up, as decided by the operating system.
Once it has determined the number of workers, and if the number of workers is larger than 1, Playwright will then decide how to split the work between workers. This decision also depends on several factors, such as the number of browsers involved and the granularity of the parallelism, which can be controlled in several ways:
- In the test spec file, you can specify whether to run the test cases in parallel. To run tests in parallel, you can invoke
test.describe.configure({ mode: 'parallel' });before your test cases. - Alternatively, you can configure it per project by setting the
fullyParallel: trueproperty. - And finally, you can set it globally in the config, using the same property:
fullyParallel: true.
Therefore, if there is more than one worker and the parallel mode is enabled, Playwright will assign test cases to each worker as they become available. This scenario is ideal because it means that each test is stateless, and the resources are used most efficiently.
> npx playwright test
Running 6 tests using 4 workers
[chromium] › example.spec.ts:3:5 › has title
Running 'has title' test in worker 0 using browser chromium
[chromium] › example.spec.ts:11:5 › get started link
Running 'get started link' test in worker 1 using browser chromium
[firefox] › example.spec.ts:11:5 › get started link
Running 'get started link' test in worker 3 using browser firefox
[firefox] › example.spec.ts:3:5 › has title
Running 'has title' test in worker 2 using browser firefox
[webkit] › example.spec.ts:3:5 › has title
Running 'has title' test in worker 4 using browser webkit
[webkit] › example.spec.ts:11:5 › get started link
Running 'get started link' test in worker 5 using browser webkit
6 passed (3.9s)
Disabling Parallelism
What if, however, the tests are not stateless? Imagine one test changes the global configuration of the app via some sort of administration page, and the configuration affects different parts of the app, like enabling or disabling features. Other tests, which may be testing those features, would be impacted and would report incorrect results. In such cases, you might want to disable parallelism.
You can disable any parallelism globally by allowing just a single worker at any time. Following the instructions from the previous sections to configure the number of workers, you can either set the workers: 1 option in the configuration file or pass --workers=1 to the command line.
npx playwright test --workers=1
Let's have a look at our test output in this case:
Running 6 tests using 1 worker
[chromium] › example.spec.ts:3:5 › has title
Running 'has title' test in worker 0 using browser chromium
[chromium] › example.spec.ts:11:5 › get started link
Running 'get started link' test in worker 0 using browser chromium
[firefox] › example.spec.ts:3:5 › has title
Running 'has title' test in worker 1 using browser firefox
[firefox] › example.spec.ts:11:5 › get started link
Running 'get started link' test in worker 1 using browser firefox
[webkit] › example.spec.ts:3:5 › has title
Running 'has title' test in worker 2 using browser webkit
[webkit] › example.spec.ts:11:5 › get started link
Running 'get started link' test in worker 2 using browser webkit
6 passed (8.2s)
Now, compare the time it took with one worker versus the time it took with four workers. It took 8.2 seconds with one worker compared to only 3.9 seconds with multiple workers. That might be an inefficient usage of resources, of course, especially if you have a large test suite and some of the tests are stateless and can be run without impacting other tests.
Tweaking Parallelism
If you want some tests not to run in parallel, but still want to utilize your workers, you can do that. Again, following the configuration options from the previous sections, you can annotate the entire spec file with test.describe.configure({ mode: 'serial' }); to have the tests run sequentially in that spec file, or use the fullyParallel: false property to run tests sequentially on the project level, or using the same property to run tests sequentially on the global level.
This means you can still split the tests between workers, but the tests would be run sequentially within a worker depending on the configuration. For example, let's set the number of workers to 4, but set fullyParallel: false globally.
> npx playwright test --workers=4
Running 6 tests using 3 workers
[chromium] › example.spec.ts:3:5 › has title
Running 'has title' test in worker 0 using browser chromium
[webkit] › example.spec.ts:3:5 › has title
Running 'has title' test in worker 2 using browser webkit
[chromium] › example.spec.ts:11:5 › get started link
Running 'get started link' test in worker 0 using browser chromium
[webkit] › example.spec.ts:11:5 › get started link
Running 'get started link' test in worker 2 using browser webkit
[firefox] › example.spec.ts:3:5 › has title
Running 'has title' test in worker 1 using browser firefox
[firefox] › example.spec.ts:11:5 › get started link
Running 'get started link' test in worker 1 using browser firefox
6 passed (3.7s)
The tests need to be run sequentially, but since each browser instance effectively provides an isolated environment, this means tests cannot impact each other between environments, and they are safe to be executed sequentially within an environment. This means setting fullyParallel: false on the global level is not the same as having workers: 1, since the browsers themselves offer an isolated environment for the tests to run sequentially. However, since we only have 3 environments (3 browsers), we cannot fully utilize 4 workers as we wanted, so the number of workers is 3.
Conclusion
In conclusion, Playwright's workers are the core of its parallelism capabilities, enabling tests to run concurrently across different environments and browsers with efficiency. Through its many configuration properties, Playwright allows you to configure parallelism at multiple levels, offering a granular approach to optimizing your test runs. Beyond just executing tests in parallel on a single machine, Playwright's parallelism can even extend to splitting work across multiple machines through sharding, significantly enhancing the scalability of your testing.
We hope this blog post was useful. For those interested in delving deeper into the world of Playwright and its powerful features, we've also recently released a JS Drop titled Awesome Web Testing with Playwright, and we also hosted Debbie O'Brien from Microsoft in a Modern Web episode.
