Skip to content

Applitools — The Automated Visual Regression Testing Framework

Applitools is a fully-packed automation framework for visual UI regression testing. Whether you’re already serious about visual UI testing and looking for an easy way to automate this type of testing for your web and mobile applications, or you’ve just started looking into visual UI testing techniques, you may want to give Applitools a try to discover the advantages this tool provides in terms of automating your visual UI tests with regression-testing at the core!

In this post, we introduce the features of Applitools. In particular, we explain how Applitools can be used with a variety of programming languages and technologies, and we shed light on the importance of visual regression testing. You will learn the unique ways Applitools facilitates the process of running regression tests, and how Applitools easily reports test results including visual discrepancies, which is something very difficult to automate effectively without the AI engine provided by Applitools.

UI Testing

With UI testing, you test the function of your web app via simulated user inputs. For instance, by using the Cypress.io testing framework, you can simulate user actions of opening a web browser, locating and clicking a button on the page, or even inputting some text into an input field. This works perfectly and can be a great aid to help test the UI side of your application by verifying the function of the pages and how they will perform in production.

However, UI functional testing is only one aspect of testing, and this alone is not enough to do a full visual test of your application. We must consider things such as changes in the layout, size, or location of the components on the page, changes in the text formatting or content, and many other visual changes that are more difficult to detect with traditional functional testing. This is especially true with responsive web applications that change layout based on the size of their viewport.

Hence, visual UI testing is inevitable and a must! Otherwise, unintended UI changes (change in CSS, content, etc.) will go unnoticed in your testing and affect your users.

Visual UI Testing

Automated visual UI testing is a form of regression testing that follows certain steps in order to validate that screens or pages have not changed unexpectedly from one test run to another. The figure below depicts a typical test case simulating a user’s actions for opening the “Wikipedia” website and searching for “Software” as a keyword.

The process of doing visual UI testing starts by creating a typical integration test case using the same tools and test suites you would normally use in executing such integration tests. An integration test is a series of actions executed on the app screens/pages, simulating an actual user.

With visual UI testing, you follow the same steps as above but with a twist:

  1. Open a browser and navigate to the “Wikipedia” website.

  2. The step above creates a new state of a “Browser opening Wikipedia website”.

  3. Take a snapshot of this state. If this is the first time the test case is running, then you will save this snapshot as a “baseline” for this step within the test case. This is also true for the other snapshot steps below.

  4. Entering the “Software” keyword into the Search-box.

  5. The step above creates a new state of “Wikipedia English website and the search-box field populated with the keyword Software”.

  6. Take a snapshot of this state.

  7. “Wikipedia” Software page opens.

  8. The steps above create a new state of “Wikipedia Software page displayed for user”.

  9. Take a snapshot of this state.

The next time you run the same test case again, upon taking a snapshot, you would compare the snapshot to the baseline and report the results. The results could be:

  • No change: In this case, the new snapshot is equivalent to the baseline snapshot.

  • Change: In this case, the new snapshot has some UI changes compared to the baseline snapshot.

The visual UI tester is now responsible for comparing the snapshots, analyzing them and deciding how to address any differences. They may accept the differences, save the new snapshot as the baseline and use it for the coming regression tests. Or, they may reject the differences and inform the developers of the bugs and issues noted by the results.

How Applitools works

Applitools uses the same approach as above by providing the tools that you can use to take snapshots, send snapshots to the Applitools cloud and run AI logic to compare the snapshots with their corresponding baselines.

Applitools provides a rich set of Software Development Kits (SDKs) for well-known test suites. For instance:

For instance, Applitools for Selenium SDK wraps the Selenium WebDriver and provides a new and enhanced WebDriver that you can use to simulate a user’s actions on a screen or page. The only difference is that Applitools is now aware of each and every action you execute on the Selenium WebDriver. This makes it easy for Applitools to capture snapshots of the results of an action executed against the WebDriver.

When using Applitools SDK in your test cases, you can ask the SDK to capture the state of the application whenever you want. This process is as follows:

  1. The SDK takes a snapshot of the browser

  2. The resultant snapshot is sent to the Applitools Server

  3. If there is no previous baseline image, the Server stores the image as the baseline for comparison in future test runs

  4. If there was a previous baseline image, the Server runs its AI engine to compare the baseline image to the new snapshot. The engine reports whether the images match, or the engine reports any discrepancies between the two images.

  5. Upon completion of the tests, a detailed report is generated on the differences between the various snapshots and their corresponding baselines.

With the recent launch of Cypress.io and Storybook SDKs, Applitools took a new and enhanced approach to visual testing — the Applitools Visual Grid. Instead of snapshot images, DOM Snapshots are uploaded to the Applitools Servers, and these snapshots are now analyzed in parallel. The result is up to 30x improvement in the speed of your visual testing suite.

Applitools Test Manager

In addition, Applitools provides a rich Test Manager dashboard to help you manage all the tests you run. You can run multiple test cases within a single batch.

For each test case, you get a list of all the snapshots taken throughout the execution of the test case. The dashboard will notify you of any test cases that are unresolved, meaning the snapshots generated by the test case might be different from those stored as baselines. The dashboard provides the tools to mark those changes as accepted and updates them as the new baseline snapshot, or rejects the differences and marks those changes as bugs for the developers to address.

The Test Manager provides a variety of features to make the process of checking and validating test case results an easy one.

The powerful AI engine behind Applitools can use one of four pattern matching algorithms to compare the snapshots to their corresponding baselines. The algorithms are:

  1. Exact: This is a pixel-to-pixel matching technique which is generally not recommended due to its strictness.

  2. Strict: This matching technique compares everything including content (text), fonts, layout, colors and position of each of the elements. This mode ignores the rendering changes that are not visible to the human. You will generally get the best results with this mode of comparison.

  3. Content: This option is similar to the strict option above except that it only performs content (text) comparisons.

  4. Layout: This option is similar to the strict option above except that it only performs layout comparisons.

You can specify the matching strategy within the test case code itself or through the Test Manager when viewing the comparison results between the snapshots.

Screenshot API

In addition to selecting and using one of the available SDKs, Applitools provides the Screenshots API in a variety of programming languages that can help you in taking snapshots of anything in your application and sending those snapshots to the Applitools Server to process and generate comparison results.

The Screenshots APIs available are:

With a Snapshots API, you can easily send snapshots to the Applitools Server, run comparisons on the snapshots, and generate comparison results. Hence, the Snapshots API could be used as a general purpose gateway to access the AI engine sitting behind the Applitools Server.

Demo

Now let’s switch gears and get our hands dirty developing a simple test case making use of the Applitools SDK for Cypress.io.

In this test case, we will simulate the process of navigating to the “Wikipedia.com” website, clicking to the English version of the website, typing “Software” in the search-box field, and finally navigating to the “Software” page on Wikipedia.

Initialize NPM on a new folder

I will start by creating a new folder and initializing it with NPM to access the package.json file. Issue the following command and follow the steps required to generate the file:

npm init

Install Cypress package

The next step is to install the Cypress NPM package with its files and start testing your source code with Cypress. Issue the following command:

npm install cypress --save-dev

This command installs Cypress package as a dev dependency onto your project.

Now open Cypress and run it locally. Issue the following command:

npx cypress open

The first time you run this command, Cypress prompts you with a warning message that it is about to add some configuration files and sample tests to help you get started. It will appear like this.

Click on Ok, got it! button to proceed and run Cypress locally. Two things happened here:

  1. A new folder named cypress was created inside your project root folder with a few sub-folders inside following the structure of a typical Cypress folder-configuration system that is used to run your tests.

  2. The Cypress Test Runner is now open and you can start playing around with the sample tests and run them.

Click on any of the tests and familiarize yourself on how Cypress runs tests in the browser.

Create our first test case

Let’s go ahead and create our first test case with Cypress. In the context of this article, I will assume you know how to write Cypress test cases. If for some reason you require additional information, you can visit the Cypress website and explore their detailed documentation by clicking on this link: Cypress Getting Started.

Let’s get going!

  1. Add a new folder named Wikipedia under the cypress/integration folder.

  2. Add a new JavaScript file named searching.spec.ts under the new folder you created.

  3. Include the following source code inside the new file

/// <reference types=”Cypress” />

    context(‘Wikipedia’, () => {
      beforeEach(() => {
        // Go to Wikipedia website
        cy.visit(‘https://www.wikipedia.org/')
      });

      it(‘Go to Software page by searching for “Software”’, () => {
        // Find the “English” link and click it
        cy.get(‘#js-link-box-en’).click();

        // Find the search-box control and type “Software”
        cy.get(‘#searchInput’)
          .type(‘Software’) // Type “Software”
          .should(‘have.value’, ‘Software’) // Check for correct value
          .type({enter}); // press enter

        // check if the element has a class  
        cy.get(‘#firstHeading’).should(‘have.class’, ‘firstHeading’);
      })
    })

Before each test case run, the code is instructing Cypress to visit the “Wikipedia” website. Our test case for this article verifies that when searching for “Software” on Wikipedia, the user is directed to the Software page there.

Notice, the code starts by finding the English link on the Wikipedia website and clicking it. Now, Wikipedia is in a state where the English version of its website is displayed to the user.

The test case locates the search-box input control, on the page, by its ID. Then, by using the powerful chaining/fluent Cypress API, it issues a command to type the text “Software” into the search-box input control. Then it makes sure the input control now holds the text of “Software”. Finally, it issues an Enter command to actually submit the search form.

The last line of code assumes that the user has been transferred to the English page on Wikipedia. It locates an HTML DIV by its ID and verifies that it has a CSS class of firstHeading.

The test case is fairly simple yet it gives you an idea on how you may use Cypress to write your functional/UI tests for your application.

Let’s run the tests by issuing the following command:

npx cypress open

On the Cypress Test Runner, locate the searching.spec.ts file and click on it.

Cypress runs the selected test and open a new instance of the Chrome (headless) browser to run all the commands in the test case.

You may go through the test case steps on the left-hand side of the page and try replaying individual steps and watch how they get executed on the fly on the right-hand side of the page.

Now let’s add some visual UI testing with Applitools.

Add Applitools Cypress SDK to the mix

Let’s start by installing the Applitools Cypress SDK into our project. For that purpose, you can issue the following command:

npm install @applitools/eyes.cypress --save-dev

The npm command above installs the Applitools eyes.cypress plugin as a dev dependency on your project. (Eyes is the name of Applitools’ product.)

The next step is to configure the eyes.cypress plugin by issuing the following command:

npx eyes-setup

The npx command above adds the necessary imports to your cypress plugins and support files. You may read additional information on how to setup eyes.cypress in full detail by clicking on this link: @applitools/eyes.cypress.

In brief, the command added the following line of code to the plugins/index.js file:

require(‘@applitools/eyes.cypress’)(module);

In addition, the following line of code was also added to the support/index.js file:

import ‘@applitools/eyes.cypress/commands’

With these packages now installed on the project, you may start enjoying visual UI testing!

Let’s amend the test case with a few calls to the eyes.setup plugin to save snapshots for the different test case states and send them to the Applitools Servers.

    /// <reference types=”Cypress” />

    context(‘Wikipedia’, () => {
      beforeEach(() => {
        // Go to Wikipedia website
        cy.visit(‘https://www.wikipedia.org/')

        // Open a connection with Applitools Servers
        cy.eyesOpen({
          appName: ‘Intro to Applitools Cypress SDK,
          testName: ‘Searching Wikipedia’,
          browser: { width: 800, height: 600 }
        });

        // Take a snapshot after the browser opens Wikipedia page
        cy.eyesCheckWindow(‘Wikipedia home page’);
      });

      afterEach(() => {
        // Close the connection with Applitools Servers
        cy.eyesClose();
      });

      it(‘Go to Software page when searching for “Software”’, () => {
        // Find the “English” link and click it
        cy.get(‘#js-link-box-en’).click();

        // Take a snapshot of Wikipedia English Page
        cy.eyesCheckWindow(‘Wikipedia English Page’);

        // Find the search-box control and type “Software”
        cy.get(‘#searchInput’)
          .type(‘Software’) // Type “Software”
          .should(‘have.value’, ‘Software’) // Check for correct value
          .type({enter}); // press enter

        // Take a snapshot of the Software Page on Wikipedia
        cy.eyesCheckWindow(‘Software Page on Wikipedia’);

        // check if the element has a class
        cy.get(‘#firstHeading’).should(‘have.class’, ‘firstHeading’);
      })
    })

The test case opens a connection to the Applitools Servers before running each and every test case using the following line of code:

    cy.eyesOpen({
      appName: ‘Intro to Applitools Cypress SDK,
      testName: ‘Searching Wikipedia’,
      browser: { width: 800, height: 600 }
    });

The call to the eyesOpen() method specifies the name of the application, the name of the test case and the browser dimensions. You have plenty of properties to include on this method, and you may check them all by clicking on this link: @applitools/eyes.cypress.

The code then issues a command to the eyes.cypress plugin to take a snapshot of the current state of the test case.

    // Take a snapshot after the browser opens Wikipedia page
    cy.eyesCheckWindow(‘Wikipedia home page’);

The eyesCheckWindow() function takes as input the name of the snapshot and sends it instantaneously to the Applitools Servers.

After running each and every test case the code also closes the connection with the Applitools Servers:

    afterEach(() => {
      // Close the connection with Applitools Servers
      cy.eyesClose();
    });

In the body of the test case, I include several calls to the cy.eyesCheckWindow() function to collect a few snapshots for the different states of the test case.

As you can see, the process of using the eyes.cypress is fairly straightforward.

Let’s run the tests again by issuing the following command:

npx cypress open

You can follow on Cypress while it is running the test case.

Let’s review the results in the Applitools Test Manager.

Review the results in the Applitools Test Manager

Now that we run our visual/UI test case with Applitools Cypress SDK, let’s explore the Applitools Eyes Test Manager and see what happened here.

To log into the Test Manager sign-in at the Applitools website.

The image shown above is what you see when you log in to the Applitools Test Manager.

You notice a single batch on the left-hand side. This is the test case that we performed. Applitools can run multiple test cases in a single batch. In this case, we only have a single test case in the batch.

By clicking on the batch the Test Manager displays the test case(s) inside this batch on the right side of the screen.

The status of the test case is “New”, meaning this is the first time the test case was executed. Therefore, there are no previous baseline snapshots to compare against.

By clicking on the test case you can see all the snapshots that Applitools Cypress SDK took during the running of the test case.

Clicking on the third snapshot, the dashboard expands the image so you can review it carefully.

In general, when viewing a snapshot in the zoomed-mode, you get the chance to see the current snapshot on the right-hand side and the baseline (if any) on the left-hand side.

If you cannot see both snapshots, then you need to instruct the Test Manager to display both as shown in this figure:

Let’s run the test case once again, except this time, before running the test, we will change the search term in our test from “Software” to something else like “Applitools”. This will imitate the effect of having a visual regression in our test results that we would like Applitools to detect.

Run the tests and then visit the Test Manager and notice the following results.

Now we can see the two batches that we ran. However, the second run of the same batch has a status of “Unresolved”. This means that the snapshots that were taken in the second batch have some differences compared to the baseline snapshots taken in the first run.

By clicking on the test case on the right-hand side of the screen you can see the following:

The Test Manager displays all of the snapshots in the test case. The third snapshot shows a difference between this one and the baseline snapshot. You may click on the third snapshot and view the differences.

You can notice the Not Equal sign surrounded by a red rectangle (The rectangle added on the snapshot above to highlight this section). The sign clearly shows that the new snapshot is different from the baseline snapshot taken in a previous run for the test case. This demonstration simulates a regression testing cycle where things change (content or anything else) and with Applitools Test Manager you can detect the changes and report them.

This was just the tip of the iceberg in exploring the Test Manager. You may learn more about Applitools and the Test Manager by visiting the Applitools website.

Conclusion

In this article you have explored writing a UI/integration test case, converting the test case into a visual/UI test case using Applitools Cypress SDK, and using the Test Manager to study and analyze the results.

There is plenty of information about Applitools that I recommend you check on their website if you are really serious about adding visual UI testing to your projects.

Happy Applitooling!

This Dot Labs is a development consultancy that is trusted by top industry companies, including Stripe, Xero, Wikimedia, Docusign, and Twilio. This Dot takes a hands-on approach by providing tailored development strategies to help you approach your most pressing challenges with clarity and confidence. Whether it's bridging the gap between business and technology or modernizing legacy systems, you’ll find a breadth of experience and knowledge you need. Check out how This Dot Labs can empower your tech journey.

You might also like

Introducing the release of Vue 3 cover image

Introducing the release of Vue 3

Back in October 2018, Evan You announced plans to start building the third version of VueJS. Featuring 30+ RFCs, over 2,600 commits, 628 pull request from 99 contributors, Vue 3's release reflects tireless ingenuity, passion, and hard work. Its release, no doubt, is a cause for celebration for all of the Vue community members who have been eagerly awaiting it. I, for one, am thrilled to share some resources that I believe will help developers migrate to Vue 3: - Vue 2 -> Vue 3 Migration Guide - Vue Router 3.0 -> Vue Router 4.0 Migration Guide - Vuex 4 - Chrome Vue JS DevTools - FireFox DevTools Here at This Dot, we have tracked Vue 3's development from the onset. We have written blog posts, published a book, hosted Vue Meetups, debuted JavaScript Marathon, shared Modern Web Podcasts, and more! Today, we’ll take a guided tour through all the material we have shared on Vue 2 and Vue 3. Blog Posts Here are our latest blog posts on Vue 3: - How to Set Up Storybook in Vue 3 - Async Components in Vue 3 - Your first Vue 3 app using TypeScript - Teleporting in Vue 3 - Content Distribution in Vue 3 - Custom Directives in Vue 3 - Vue 3 Composition API, do you really need it? You may access the remaining blog posts at This Dot Labs Blog. A Complete Guide to VueJS In April 2020, This Dot released A Complete Guide to VueJS by Bilal Haidar. This book is a precise, and detailed resource for learning VueJS, and highlights some top courses for Vue and JavaScript. Most importantly, it gives a brief introduction to almost all the new features in Vue 3. Grab your own copy for free! Vue Meetups We have hosted more than 10+ Vue Meetups in the past year with dozens of speakers, including Evan You, other Vue Core team members, and Vue developers interested in the future of the framework. To watch past meetup recordings, follow this link to get access to all the meetups on one page. JavaScript Marathon This Dot's team delivered six free live Vue JS tutorials during the JavaScript Marathon. Here’s a list of all the VueJS live sessions recordings: - 1 Hour to Learn Vue - Master State Management in Vue with VueX - Master PWA in Vue - Learning Unit Testing in Vue - Pro Tips on Using AWS with Vue - Debugging Vue: Quick Tips and Tricks Modern Web Podcasts This Dot's team delivered more than 40+ podcasts in the last two years. Here’s a link to the Vue JS Podcasts: - S07E1 Modern Web Podcast - Introducing Vite - Evan You’s new project + Vue 3 Updates - S06E12 Modern Web Podcast - Vue 3, Code Education, & the Vue Community - S06E4 Modern Web Podcast - Vue Updates with Chris Fitz, Jake Dohm, and Rob Ocel The Future of Vue The team at This Dot is hardly finished tracking Vue's progress, as well as the progress of many other web based technologies. To join us on our technical exploration journey, be sure to follow This Dot Media on Twitter! If you have specific questions about how to begin your Vue 3 migration, or have general questions about Vue 3, don't hesitate to reach out to us at hi@thisdot.co....

Testing a Fastify app with the NodeJS test runner cover image

Testing a Fastify app with the NodeJS test runner

Introduction Node.js has shipped a built-in test runner for a couple of major versions. Since its release I haven’t heard much about it so I decided to try it out on a simple Fastify API server application that I was working on. It turns out, it’s pretty good! It’s also really nice to start testing a node application without dealing with the hassle of installing some additional dependencies and managing more configurations. Since it’s got my stamp of approval, why not write a post about it? In this post, we will hit the highlights of the testing API and write some basic but real-life tests for an API server. This server will be built with Fastify, a plugin-centric API framework. They have some good documentation on testing that should make this pretty easy. We’ll also add a SQL driver for the plugin we will test. Setup Let's set up our simple API server by creating a new project, adding our dependencies, and creating some files. Ensure you’re running node v20 or greater (Test runner is a stable API as of the 20 major releases) Overview `index.js` - node entry that initializes our Fastify app and listens for incoming http requests on port 3001 `app.js` - this file exports a function that creates and returns our Fastify application instance `sql-plugin.js` - a Fastify plugin that sets up and connects to a SQL driver and makes it available on our app instance Application Code A simple first test For our first test we will just test our servers index route. If you recall from the app.js` code above, our index route returns a 501 response for “not implemented”. In this test, we're using the createApp` function to create a new instance of our Fastify app, and then using the `inject` method from the Fastify API to make a request to the `/` route. We import our test utilities directly from the node. Notice we can pass async functions to our test to use async/await. Node’s assert API has been around for a long time, this is what we are using to make our test assertions. To run this test, we can use the following command: By default the Node.js test runner uses the TAP reporter. You can configure it using other reporters or even create your own custom reporters for it to use. Testing our SQL plugin Next, let's take a look at how to test our Fastify Postgres plugin. This one is a bit more involved and gives us an opportunity to use more of the test runner features. In this example, we are using a feature called Subtests. This simply means when nested tests inside of a top-level test. In our top-level test call, we get a test parameter t` that we call methods on in our nested test structure. In this example, we use `t.beforeEach` to create a new Fastify app instance for each test, and call the `test` method to register our nested tests. Along with `beforeEach` the other methods you might expect are also available: `afterEach`, `before`, `after`. Since we don’t want to connect to our Postgres database in our tests, we are using the available Mocking API to mock out the client. This was the API that I was most excited to see included in the Node Test Runner. After the basics, you almost always need to mock some functions, methods, or libraries in your tests. After trying this feature, it works easily and as expected, I was confident that I could get pretty far testing with the new Node.js core API’s. Since my plugin only uses the end method of the Postgres driver, it’s the only method I provide a mock function for. Our second test confirms that it gets called when our Fastify server is shutting down. Additional features A lot of other features that are common in other popular testing frameworks are also available. Test styles and methods Along with our basic test` based tests we used for our Fastify plugins - `test` also includes `skip`, `todo`, and `only` methods. They are for what you would expect based on the names, skipping or only running certain tests, and work-in-progress tests. If you prefer, you also have the option of using the describe` → `it` test syntax. They both come with the same methods as `test` and I think it really comes down to a matter of personal preference. Test coverage This might be the deal breaker for some since this feature is still experimental. As popular as test coverage reporting is, I expect this API to be finalized and become stable in an upcoming version. Since this isn’t something that’s being shipped for the end user though, I say go for it. What’s the worst that could happen really? Other CLI flags —watch` - https://nodejs.org/dist/latest-v20.x/docs/api/cli.html#--watch —test-name-pattern` - https://nodejs.org/dist/latest-v20.x/docs/api/cli.html#--test-name-pattern TypeScript support You can use a loader like you would for a regular node application to execute TypeScript files. Some popular examples are tsx` and `ts-node`. In practice, I found that this currently doesn’t work well since the test runner only looks for JS file types. After digging in I found that they added support to locate your test files via a glob string but it won’t be available until the next major version release. Conclusion The built-in test runner is a lot more comprehensive than I expected it to be. I was able to easily write some real-world tests for my application. If you don’t mind some of the features like coverage reporting being experimental, you can get pretty far without installing any additional dependencies. The biggest deal breaker on many projects at this point, in my opinion, is the lack of straightforward TypeScript support. This is the test command that I ended up with in my application: I’ll be honest, I stole this from a GitHub issue thread and I don’t know exactly how it works (but it does). If TypeScript is a requirement, maybe stick with Jest or Vitest for now 🙂...