Mabl AI Machine Learning Test Automation

A few months ago, I spoke with Dan Belcher, co-founder of Mabl, about AI based testing tools. Mabl was still in development at the time, so I was excited to hear that the Beta version is now available.

What is Mabl?

Mabl is one of the newer tools in the market that really focuses on the hot trend of using AI and machine learning to help folks with their testing efforts.

How Mabl AI Automation Works

In a nutshell, to get started you simply point Mabl to your application. It will then crawl your app’s screens and begin to run default tests that are common for most applications. Along the way, it also uses machine-learning algorithms to improve test execution and defect detection. Also using the Mabl trainer chrome extension you can train Mabl to learn journeys in your application which describe the functionality Mabl will test.

How is Mabl Different from Selenium?

For one thing, it doesn’t rely on brittle element indicators to interact with your application elements. In fact, this was one of the reasons Mabl was created.

After speaking to a number of companies, the common theme the co-founders of Mabl found was that almost everyone they spoke was struggling with Selenium. Most of the issues were due to the element identification not being reliable, which was leading to test failures.

As most of you know, Selenium is pretty rigidly tied to the specific elements on your applications’ front end. Flakiness often arises when you make a seemingly innocent change to a UI (changing an ID for an element or putting it in a different div on the front end, for instance).

Unfortunately, many Selenium tests end up being unable to find those changes, leading to test failure.

Mabl takes a different approach in that it has a smarter way of referring to front-end elements in your test automation.

So the types of changes that will break your tests in Selenium will not break your Mabl tests. Mabl understands the difference between what might be a very small change to an element and a new feature or broken front end. 

Mabl Testing Insights

Besides using AI and machine learning techniques to interact with your application without relying on elements, Mabl also provides what they call “insights.”

An insight is a behavior in your application that Mabl flags as an area of interest that might be important for a development team to know about. Insights are identified by the tool based on what's changed between two test runs for a given application and then presenting that to the engineer or QA tester.

This info is simply laid out and shows you–at a very specific point in the test–exactly what the front end looked like using captured runtime screenshots. You can see what was in the JavaScript console at that time. Future releases of Mabl will also include the ability to see what was visually different about that screenshot versus the previous screenshot.

This type of functionality makes it easy to troubleshoot failing tests.

Analysis pipeline

When it comes to the typical test output you would expect with an automation tool, Mabl is different in that is uses an Analysis Pipeline to analyze your application behavior over time.

As you run tests in Mabl, it collects rich data that spans not just test output logs but also HTML data, images, screenshots of the state of pages, and errors coming out of Chrome and other browsers.

All this test data is fed into a machine-learning pipeline.  This is what is called the “training model.” This allows the tool to look at all the different datasets of what the application is doing, and what is normal about the application for each step in each test.

So, unlike traditional functional test automation scripts that use historic run data, AI-based tests are able to adapt to learning from previous runs. This means that the quality of the models improve automatically over time the more Mabl executes your tests.

By capturing as much data as possible it really is using the data for training leveraging the machine-learning models, which in turn give you better insights and more reliable tests.

So what are some examples of an insight?

Machine Learning Insight Examples for Analysis Pipeline

One example of an insight is the performance of your application.

Mabl measures timing for every page, every image and every component, each time you run the test.

This then feeds all this data to the analysis pipeline and training models of what typical performance looks like for all of those pages and components and test.

Then Mabl is able to make predictions with every test run of the range you can expect a page to load in or a test runs normal run time.

If it becomes obvious that the actual load time is outside of the predicted range, it will generate an insight that will point out, for example, that your login page is significantly slower than it was before the last deployment.

Automation Timings

Other elements that cause automated tests to fail randomly are timing issues–and of course, Mabl has that handled as well.

As automation engineers, false positives drive our teams crazy. With Mabl you can just click on an element and train it to actually reduce sensitivity around that page, which in turn will feed the machine-learning model once again to be explicitly less sensitive about any dramatic timing changes in that page or element.

How will AI Tools Change Automation?

So how will AI automation tools like Mabl change the automation testing landscape?

Dan believes that from an automation perspective it's really going to be more about enabling people who have been running tests to determine whether things are working to start looking at quality with a different definition—for example: ease of use, performance, and other things that affect user satisfaction.

Using an AI tool like Mabl will allow engineers to spend more hours building capabilities for their end users and fewer hours dealing with test automation.

The hope is that with machine learning there will be measurable impact on allowing developers to spend more time developing features for their customers.

Another way of putting it is that AI testing may allow us to adopt a more product- and user-focused approach to their world. That’s not so bad, is it?