Applitools What is Visual Validation Testing

Adam Carmi is the co-founder of Applitools, a company that specializes in visual validation. Last week I was lucky enough to speak to him about visual validation testing on an episode of Test Talks.

I think it was a great episode and many of the things I touch on in this post you can listen to in detail in –> Episode 43 <–.

Why is Visual Validation so hard?

If anyone has tried doing any sort of visual testing using tools like Eggplant and UFT Insight, you know how hard it is to make these types of tests reliable. Sometimes the tests are so fragile you can only run them on the same machine they were developed on to avoid flaky tests. There are many reasons for this, but it's mostly due to pixels being slightly off from one browser or OS.

Applitools eye is different in that it was developed from the ground up for visual validation and its sophisticated algorithm was designed to handle many pixel issues that most other image-based testing tools have a hard time handling.

Applitools allows you to find and automatically detect all the visual bugs to validate the visual correctness of your application.

What is Visual Validation?

Before we get into it, you might be asking “What is Visual Validation testing?”

Visual testing is a quality assurance activity that is meant to verify that UI appears correctly to users. Many people confuse that with traditional, functional testing tools like Selenium and QTP/UFT which were designed to help you test the functionality of your application through the UI.

When we're talking about visual testing we want to make sure that the UI itself looks right to the user and that each UI element appears in the right color, shape, position and size.

We also want to ensure that it doesn't hide or overlap any other UI elements. In fact, many of these types of tests are so difficult to automate that they end up being manual tests.

Applitools was designed to help users create highly maintainable, visual validation tests on a very large scale.

Use the right Test Tool for the right job

One thing to keep in mind is that as an automation engineer, you should always look for the right tool to do each job. Not everything was meant to be a Selenium or QTP test.

Think of Applitools as another tool that you can use in conjunction with Selenium and QTP to handle the complicated visual types of testing you need to get done, and leave the more functionality-type testing to the tools that where designed to handle that, like Selenium.

These tools work together; it's not either/or.

At first, Applitools looked at all the existing testing tools that were in the market and focused on the functionality that they felt was not adequately covered.

The answer was very clear that Selenium and QTP were (and are) very good for simulating user actions like clicking a button and entering key strokes, but not so good with handling visual validation.

As a result, Applitools decided to make their core specialty handling complex visual validation in a really robust, scalable way.

So rather than just creating another functional testing tool, they decided to focus on enhancing their unique image-based capability and just allow it to be connected to any of the other frameworks/test tools — or even just work with plain images — even if there is no test automation framework.

Applitools integrates with all the main test tool vendors and development languages.

How Applitools works

Applitools is a Cloud service and can also be installed on premise, but it's basically a server with a REST API. As long as you can do http and upload images you're good to go.

There are also a bunch of SDKs for most of the programming languages that allow you to take images and upload them for visual validation.

It doesn't matter where these images come from and how you get them, whether from a web mobile device, desktop application, web browser, or even taken from Photoshop — you just need to upload it for visual validation.

For the leading testing frameworks like Selenium and UFT, Applitools created more sophisticated SDKs that actually hook into your test and abstract the grabbing of images.

For example, with Selenium, Applitools can automatically scroll the image and stitch it together to get a screenshot of the entire page (even when using Chrome). You also can just check an inner frame and Applitools will scroll and check it for you.

It also can be used to help your test scripts know when a web page has fully loaded and stabilized.

For example, if you have a page that can take a long time, you might have to add all kinds of waits within your scripts to try and handle synchronization issues with long-loading pages. But with Applitools, since you already have a baseline image that tells you when the application is ready, you can use it as a more reliable wait method.

The SDK tries consistently to take screenshots and compare them.

If your slow loading page is ready after one second, the test will continue but if it takes 20 seconds to load it will wait the full amount of time.

This is cool, because if you are developing against Chrome or Firefox elements tend to load faster on those browsers. If you rerun against IE, the test will sometimes fail because someone didn't think to wait for the page or an element to be ready.

So, you could use Applitools in your existing Selenium scripts just as a synchronization point to ensure before you continue that the page is ready for input.

Because it's a visual check, you don't need any special code to tell it what browser you are running against. You just add a single validation point and the SDK takes care of the synchronization for you.

Collaboration Tool

By using Applitools, you also get some extra features that are a side effect of using the tool. One of those features is a dashboard that can be used almost like a collaboration tool.

Since you have an Applitools server any one with permissions can access and view the playback of tests that include the Applitools visual checkpoints as well as all the navigation actions (like clicks and keyboard strokes) played against them.

This allows users that don't normally have access to code (and don't understand code), like project managers, to have a place to view the UI of the application on a daily basis, see what has changed, and be able to give immediate feedback to developers.

That means they don't have to wait for the sprint to end to have a demo of the changes or new functionality. Also, they needn't have access to the latest build of the application under test — all they have to do is go in and look at the tests and view the differences that were found as well as see what happens whenever the tests run.

How is Applitools different from tools like Eggplant and UFT?

The main difference is that tools like Selenium, UFT, QTP, and Sikuli were designed to solve a different problem than visual validation; they were designed to allow you to click a button, enter text or validate some output that you cannot otherwise do by writing code.

There's a big difference between finding a button and clicking it and actually validating that the full application page looks right. There's a big difference both with the image processing technology that is needed and the image capturing and maintenance needed for visual tests.

Basically, the majority of these tools need to capture an image but because the image does not always render the same every time. As a result, you have to specify a threshold, like 80%, to ignore some of the differences so that you can still find the button.

For automating something that will test some functionality this is normally good enough, but it's not enough to validate something that has an 80% threshold.

Also, most of these tools did not do anything innovative with image-process technology. They basically just used existing building blocks that are textbook algorithms for image processing and exposed them from the UI. A tester actually has to configure the way in which they want to match images.

You may be trying to automate your test scripts, and you're suddenly being asked wither you want a Sobel edge detector or a Canny edge detector. Of course, if you're anything like me you have no idea what this means, but you still need to choose one.

You probably just go with whatever seems to work, with no understanding of what any of the threshold settings and configurations really means. Because of that lack of understanding, you ultimately get flaky tests.

The problem is even worse in that none of these algorithm and image-processing techniques are suitable for actually solving the problem. They are general-purpose algorithms, and are not really a good way to prevent false positive and bug misses in your test script.

Applitools is different because they have invested years in research and development, creating awesome image-processing algorithms from scratch. It is a total image-process stack created for the sole purpose of being able to compare images that were originally rendered by a computer's graphics card.

Applitools also simplifies the testing flow you follow to capture and modify images and create visual validation scripts. With functional test tools, if you collected a thousand images and then your application changes, you have to manually capture all the images again.

The problem gets worse when you have to run your tests against multiple environments, browsers, and configurations because you then have to take duplicates of those images. At that point, your image-based testing effort has become a maintenance nightmare.

With Applitools, declaring a visual validation point automatically performs that capture, and every new run automatically captures new images — so it's easy to update the baseline for subsequent runs.

Applitools also has many options that allow you to easily run your test against very dynamic applications and ignore the things (like ads) that your tests do not care about. You're able to simply validate the layout of a page and ignore any dynamic elements that make up that page.

Listen for more info on Applitools

For more info on Applitools, be sure to listen to Episode 43 of TestTalks, on which Adam goes into detail on not only the things covered in this article but many other topics that will make your visual validation testing efforts successful.