Unlocking end-to-end testing for all with mabl

A recent *study showed that 75% of organizations use a combination of manual and automated testing.

This was a big difference from 2010 when I started my blog, which is dedicated to helping folks move toward more automation for their testing efforts.

The number back then was probably around 5-10%.

Even though the number of companies investing in automation has grown substantially, I still see a large gap.

It seems that most of the engineers I speak with are still focusing solely on automated functional tests.

It’s a great start, but many still miss a big opportunity.

INDEX
End-to-End Automation in DevOps
Looking at mabl
Low Code Test Automation
Machine Learning
Automation Testing Auto Healing Feature
Machine learning to Replace Testers?
Non-Functional Test Coverage
Quality Engineering Mindset
Reporting and Machine Learning-Enhanced Insights
Unified Platform
Unified Runner
Developer Friendly
Automation Innovation

End-to-End Automation in DevOps

I believe that if a process can be automated in your software development lifecycle, then it should be.

Many miss out on opportunities to automate in the development, build, and production phases of their pipelines.

That’s why I believe more automation engineers need to start pushing toward end-to-end testing, which includes not only functional automation but also automation for things like:

• Performance testing
• Accessibility testing
• Production reporting
• And the customer experience

And on this last point, I see opportunities for teams to focus on the user.

After all, we’re testing because we want to create better experiences for those who use our app.

Testers are in the best position to lead this; they spend more time than anyone exploring their app.

I’m seeing more automation tooling vendors adding functionality around these areas to expand automation coverage across functional and what was generally considered not automatable “non-functional” tests.

Looking at mabl

For instance, let’s look at mabl.

When I first spoke to the folks at this firm, they were still in beta mode. Their first focus was on functional UI automation testing.

Over the years, however, I’ve seen them evolve into more of a unified platform for all things software testing.

When I spoke to mabl’s founder Dan Belcher recently, he told me that when they started the company, they only focused on browser testing. Over time, however, they heard from many people that traditional automation testing was hard—especially for folks who didn't have a programming background.

So, they created a low-code approach to end-to-end test automation.

Low Code Test Automation

You might ask, “How realistic are low-code solutions, anyway?”

When I speak with other automation specialists like Paul Grossman, they say it's viable. When done right, low code is handy because it optimizes your time.

For example, when working with a low-code tool, you don't necessarily need to learn an arcane, complex language like Java, JavaScript, Python, or Ruby. Nor do you need to spend time figuring out how to implement your test. You can generally navigate your app like a user and record those steps as a test.

If you want to click on an Add to Cart button, you simply click the Add to Cart button, and that action is recorded as a step in your test.

In theory, this allows folks who might be slightly turned off by programming to contribute to the test automation efforts. It can unlock many other team members who understand the users well enough to create, run and manage efficient, realistic, and reliable test automation scenarios.

For those with coding experience, low-code solutions still allow for a level of control through JavaScript when these solutions don’t support something out of the box.

So using a low-code test automation solution like mabl has the added benefit of better test coverage at the user experience level— not only for functional tests but also for non-functional tests.

Another way to grow coverage has to do with machine learning.

Machine Learning

What’s so powerful about using a solution like mabl is that they run primarily on Google Cloud, allowing them to observe a lot of information as tests execute and process them quickly and efficiently.

Running like this allows mabl to handle all the test orchestration complexity. If you're writing scripts and managing a framework on-prem, managing the infrastructure and analyzing traces, metrics, visual information, logs, accessibility, and test output is a big challenge.

It’s a pain most testers have to deal with if you're rolling your own stuff.

But if you're running in the Cloud, there are lots of good services those Cloud providers give you to help open up more testing data.

Having all this data also leads to more reliable tests.

A great example of that is auto-healing.

Automation Testing Auto Healing Feature

Mabl’s auto-healing has built-in heuristics with different weights for different attributes, facilitating the backtesting of any changes to those heuristics to determine whether specific changes will lead to more reliable tests.

The team at mabl also told me they work with their customers directly to fine-tune their ML engine based on their architecture.

FYI—according to another recent study (buildkite), 59% of developers deal with daily, weekly, or monthly brittle tests.

Think about how many times fewer brittle tests would save you and your team.

Another example of intelligence helping testers is around waiting for elements to display. Usually, cloud tests execute pretty quickly. Sometimes so much faster that the tests run faster than your application has time to load. To avoid issues, testers typically add fixed waits that can add up over time.

Mabl learns the behavior of your system and waits only for the time necessary – executing the tests exactly when the elements are ready. For example, instead of waiting for 5 seconds by default, it may wait only half a second because it knows that historically that’s the longest it took for the element to render. This speeds up execution and helps to reduce false negatives.

Machine learning to replace engineers?

A major misconception about machine learning (ML) is that it will replace software engineers, but it’s not about replacement; it’s more about augmenting.

Using insight produced by machine learning requires a team to decide how to move forward.

Let’s be honest: I’ve been involved with automated testing for almost 25 years, and sometimes it’s still a struggle.

Even more nowadays, when you throw your whole team into the mix.

I’ve seen team after team trying to keep pace with functional quality as the pace of development velocity has increased. And there’s always the desire to extend testing for non-functional quality, but never enough time.

So, suppose you leverage more advanced AI machine learning techniques over time. In that case, it should yield you some core automation coverage—enough to allow you to focus on higher-value activities like:

• UX
• Strategy
• Collaboration
• Performance
• Ensuring your software is accessible

You want to allow testers to spend time on these things instead of the reactive stuff on which many teams have been stuck.

How should teams approach these new areas, such as accessibility and performance?

Non-Functional Test Coverage

Mabl has gradually added other features to help make it easier for someone to create end-to-end tests.

For instance, over the past few years, they’ve added performance insights and accessibility checks to be built into those low code, end-to-end tests. This helps teams get a bigger bang for their buck, just reusing the functional tests they already run.

Dan mentioned that numerous mabl customers reported that after building out all their tests in mabl, they had to recreate them in a different tool to cover things like accessibility testing.

Baking this feedback into mabl with those requested features allows users to leverage their existing functional tests and perform other critical, non-functional checks.

As I mentioned earlier, this is a trend I’ve been seeing other vendors follow as well.

Another trend I’ve seen popping up is how all these components come together into a quality engineering concept.

Quality Engineering Mindset

Three concepts make up quality engineering.

1. Quality Engineering Focused Team

For one, a quality-engineering-focused team is building quality from the minute a developer begins working on a branch locally through to production. In contrast, the traditional QA team sometimes receives a deliverable from the development team ready to be tested.

Mabl offers functionality to aid the development build stage, like offering a CLI, which we’ll cover later in this article.

2. Coverage

The second reason is that a quality-engineering-focused team considers coverage much more ambitiously.

They want to have enough automated test coverage that they can truly trust the test coverage. The difference between feeling like you have excellent test coverage is between being able to deploy automatically versus needing to manually verify before you feel that level of comfort.

Once again, mabl has advanced test coverage reporting that can analyze production and notify the team of any potential gaps in their coverage. This is done by leveraging insights produced by machine learning of all the data produced in CI/CD.

3. Data

This brings us to the third area—data.

Quality-engineering-focused teams are measuring, changing their processes, and measuring again to see what's going to improve. They're also using that same data to identify issues with non-functional requirements, whether performance, accessibility or otherwise.

I believe those three pillars are incredible coverage, building quality throughout the entire pipeline, and then using data to improve your work and non-functional quality.

Reporting and Machine Learning-Enhanced Insights

Since data is critical to quality engineering and great testing practices, let’s look at how data and machine learning play together.

Getting actionable insights into the hands of management is probably the conversation I have most often with CTOs.

As many companies are going through digital or DevOps transformation, they want to know what KPIs they should use for each of their teams and how to roll them up to help look at the initiative overall.

As discussed, an enormous amount of data can be captured when tests are executed in the cloud and helps to bubble up insights you can’t get any other way.

Having this type of data at your fingertips helps you understand what's been done related to a release. For instance, if you’re developing in a sprint, you need to know:

• How many of my tests have been executed?
• What is the pass rate or failure rate?
• How many of the tests have been updated?
• Have I added new tests during this release?
• Is my coverage sufficient?

Mabl not only helps you understand your test coverage; they take it one step further.

Teams can now correlate test coverage with usage coverage as well.

You can see your most frequently accessed pages, know how many new pages are in your application, and if you have adequate coverage over those.

As I’ve mentioned, a lot of data is curated inside of Mabl, but they can also stream that data to BigQuery, Google's version of a data warehousing solution.

Many people use that to connect to custom dashboards tailored to their team's requirements.

Unified Platform

Dan told me another lesson learned at mabl was that customers missed the opportunity to catch issues earlier by running their end-to-end tests only in later stages like a staging environment.

What they learned is that it’s almost too late.

It’s best to get a good amount of testing done before the branch is merged with your main branch. That's when it gets on the train that's destined for production.

And so, to ensure that you get good testing coverage from an automated perspective before merging, developers must validate them locally or inside of CI.

To address this, mabl created a local runner that developers can use to run those tests before they even put them up for review.

This is what mabl calls a Unified Runner.

Unified Runner

The Unified Runner allows you to use the same technology to run tests locally and then in the Cloud.

When you know that Cloud-based and locally run tests produce the same results, it gives you the confidence to release your software without fear of hidden surprises when going from one environment to another.

The Unified Runner easily incorporates your test automation into your development pipeline, allowing developers and QA engineers to test effectively while quickly releasing high-quality product updates.

To that end, mabl developed CI tool integrations and a CLI that can execute tests headlessly before it gets deployed to some environment.

Geeking out on the creation and running of the automated tests is fun, but how else can you ensure devs get involved in testing efforts?

Developer Friendly

Collaboration is a core tenant of DevOps. The same goes for testing.

To succeed with testing in Agile and DevOps environments, entire teams need to collaborate on quality – including developers.

While I’m seeing more teams engage their developers in testing activities, what I hear from teams is that work is letting developers test in their existing workflow.

The CLI I discussed is a great example.

When developers can trigger tests directly in their source control or CI tools, the chances of adoption are much higher.

With mabl’s branching and version control capabilities, mature teams can update or create new tests on a specific branch and promote those tests along with the code so that tests are ready to work with newly released features.

Gathering all this data in the cloud makes it easy for teams to see what’s happening across their application.

When more testing activity happens early in development, everyone can still access the test results. Developers can easily see what is already being tested, and QA can help guide developers when they see a coverage gap.

And when bugs inevitably reach production, cloud solutions know the exact step a test failed. Solutions like mabl integrate directly with issue tracking solutions, like Jira, to create issues directly from your test failure, including all data needed to get to the root cause quickly.

Automation Innovation

Vendors like mabl are stepping up to the challenge of modern software development by providing tools that can help you and your team(s) keep up with the rapid pace of software development.

Even for enterprise companies that traditionally embraced on-prem environments, vendors like mabl are SOC2 compliant and offer options to test internal applications secured behind firewalls securely.

If you’re struggling with automation using a more traditional approach, it may be time to test other solutions.

Mabl has a 14-day free trial, so give it a go and let me know how it worked for you.