You should always use the tool that’s best for your qa teams situation and needs. There is no perfect tool for everyone.
An example of this came up during this year’s Automation Guild conference when an interesting conversation was started regarding Galen Framework during Adam Carmi’s session.
Some folks wanted to learn more, and asked how Applitools differs from Galen Framework. Luckily, I was able to ask the creators of Applitools (Adam Carmi) and Galen (Ivan Shubin) how they would describe their tools and to discuss how they compare against one another. They graciously accepted.
What is Visual Validation Testing?
Before you we look at these tool awesome tools if you don't know what is Visual Validation Testing make sure to check out my mini course video:
Here’s what I found out:
Ivan Shubin, Creator of Galen Framework
In my TestTalks interview with Ivan he mentioned that there are two components to Galen Framework. There is an image comparison, of course, but that is explicit. You have to specify that an element should be verified using an image comparison against a specific image file, and then only the area of that element would be checked.
Galen was originally built without image comparison. It simply uses the location of the element. When Galen tests the layout, it first fetches the locations of elements, like left and top. It fetches their size. From there, it already knows the boundary box of each element. During the test case execution it checks the boundary box and nothing else.
Galen Framework is not a proper visual testing tool
That’s why Galen Framework is not a proper visual testing tool. If you don't use image comparison, you don't really check the pixels, and you don't check what is rendered.
Galen is more like a tool for developers to quickly verify that their layout is not broken.
Galen is Awesome for UX engineers and Developers
Ultimately, Galen is not a tool for visual testing. It has an image comparison feature and is very flexible, but it's also not easy to set up. If you want to quickly check pixels, there are a lot of great tools like the ones from Applitools. They do it really well.
Galen is more suited for UX engineers or developers.
Adam Carmi, Creator of Applitools
In Adams Automation Guild Q&A session he said that although Galen Framework is an awesome tool, there are some major differences between it and Applitools.
First of all, Galen allows you to write a specification of what you expect to see in the page. This implies two things.
Galen doesn't Test Everything
But it doesn't test everything; it only tests what you specify.
So if you have some rules that you want to know that are being met, or requirements that should be made to your page, it’s a great way to formalize the test so it will check those rules.
However, it won't cover 100% of everything that is visible on the screen. If you want to do that, you’ll have to write a specification. That specs has to be is as detailed and completely tied with the action implementation of the page itself. Which, of course, it will never do because it's too much work.
Doesn't Test what is being shows to the user
The second difference is that Galen Framework checks the DOM. It actually checks what you expect the browser to show. It doesn't really tell you what the browser did show.
So, for example, you could have a new version of the browser that just came out and has a bug. You might have the same DOM but different browsers shows it differently. Don’t you want your test to tell you so you can fix your code to match this difference in the browser? It doesn't have to be a bug in the browser, of course — it could just be a different interpretation of the W3C standard. It doesn't matter what the reason is; it could still mean that something is wrong with the way your site is shown.
So once again the second difference is that you don't really test what is being shown to the user; you're testing what the browser is expected to show to your user.
Galen Framework is hard to use to validate large sites
The third big difference is one that all the customers that I spoke with told me why they stopped using Galen. Galen Framework is really nice if you have a few pages that you want to validate. But if you have a large site with dozens or hundreds of those pages it becomes much more difficult. Complexity also adds up if it's a responsive website.
In that case you’d have a different DOM for each of the different form factors and screen resolutions. And what happens if you have to deal with localizations? That will also cause you to have different layouts for those different languages. You may have different roles that go in as an admin as opposed to a normal user, and each one may have its own unique role.
In this type of situation you’ll end up having to write a huge amount of specifications.
You also might make mistakes in the specifications. So you don't even sure, maybe you have a mistake in your testing your own thing, or your not even testing at all. It eventually ends up being non-scalable.
Conversely, when working with enterprise testing tools like Applitools, you simply run the test, and you can see the images. You can immediately tell that they're fine, and they give you 100% coverage of what you see.
So, if you want to run at large scale, obtain more coverage and be more accurate in your test, using visual validation tools like the ones made by Applitools will take you much further.
Which Visual Validation Tool Should you Choose?
So as you can see both tools are great choices depending on your needs! Make sure to check them both out and let me know what you think.
Also check out my resource The Top 21 FREE Visual Validation Tools for Testers
FYI: Clayton Neal left a comment in LinkedIN that I thought you might find helpful: “Nice 👍 post. One other factor is cost. Galen although layout testing is free! And free with some limitations may fit some people's budget. Visual testing I have seen comes with maturity, often the first focus is functional automation, measuring test coverage to manual tests. Pragmatically no one writes manual tests for fonts, colors or layout. And as such can be a harder sell to get visual testing when it doesn't increase % test coverage. Would like to hear others challenges in selling visual testing to their business sponsors.”
