If you’re a software tester you’ve probably been hearing a lot about autonomous software testing lately.
But how much of it is hype, and what should you pay attention to?
I’m here to help.
This article was created using insights from my latest interview with automation expert Tobias about the Myths of Autonomous Software Testing.
I’ve also included information taken from two sessions at a previous Automation Guild conferences from software developer Gil Tayar and automation expert Andy Knight.
So, read on to get real, expert info on what autonomous testing actually is, as well as how it differs from traditional automated testing.
We’ll also do some myth busting along the way.
What is Autonomous Testing?
Let’s get straight to the heart of the matter and start with some definitions.
The best way I can describe it is that autonomous testing is the use of artificial intelligence (AI) and machine learning (ML) to automate the entire testing process. So, from test case generation to execution and analysis.
Unlike traditional test automation, which requires humans to write and maintain test scripts, autonomous testing aims to create self-learning systems that can adapt to changes in the application(s) under test.
Andy Knight describes it as the next evolutionary step in efficiency with testing.
Tobias Müller, co-founder of TestResults.io agrees that autonomous testing is the future of software testing but, he thinks that we are not quite there yet. To make things even worse, most of the “autonomous testing” solutions on the market today are just wrappers around existing test automation frameworks like Selenium and Appium.
So, what should you believe? Where should you go to get a straight answer?
Well, read on to find out more.
Key Components of Autonomous Testing
Based on my conversations with other testing experts, they all seem to agree that to achieve true autonomous testing, a system needs to have the following four components:
- Vision: The ability to perceive and understand the application under test, like how humans use their eyes to interact with software
- Learning: The ability to learn from past test runs and adapt to changes in the application, using techniques like machine learning and neural networks
- Reasoning: The ability to make decisions about what to test and how to test it, based on the application's behavior and the testing goals
- Execution: The ability to interact with the application and execute tests, similar to how human testers perform manual testing
That’s a tall order, and as you will see we are still a long way off from true autonomous software testing. But even so, what are some of the benefits of what we can currently achieve with AI?
Chat with Joe Bot: Your AI Testing Guru
Benefits of Autonomous Tests
The main benefit of autonomous testing is increased efficiency and coverage.
By automating the entire testing process, autonomous testing can:
- Reduce the time and effort required for testing
- Increase the number of test cases that can be executed
- Faster release cycles and higher quality software
Another benefit of autonomous testing is what many have been touting about automation testing in general: improved reliability and consistency.
Unlike human testers, who may make mistakes or have biases, autonomous testing systems can execute tests consistently and objectively, reducing the risk of human error.
Note that this does not replace testers, but rather assists them.
So, how does one move from manual testing to this state?
From Manual to Autonomous Testing
The journey from manual testing to autonomous testing can be broken down into five stages. One great example I’ve heard was from Gil Tayar’s Automation Guild session on “Not Only Cars: AI, Please Test My Apps” during which he compared those stages to the different levels of autonomous driving:
- Level 0: No automation, all testing is done manually
- Level 1: Driving assistance, where AI assists with test validation but humans still drive the tests
- Level 2: Partial automation, where AI can drive some tests, but humans are still in charge.
- Level 3: Conditional automation, where AI can drive most tests but may need human intervention in certain situations
- Level 4: High automation, where AI drives all tests in most situations but may have some limitations
- Level 5: Full automation, where AI can test anything, a human can, without limitations
Gil believes that most companies today are at level 1 or 2, with some reaching level 3.
Achieving level 4 and 5 will require significant advances in AI and ML, so you needn’t worry about being replaced anytime soon.
But how do we get to level 5 (if it’s even possible)?
Phases of Autonomous Testing
Implementing autonomous testing typically involves three phases:
- Learning: The autonomous testing system learns the application's behavior by observing users or analyzing data from production.
- Exploring: The system explores the application to discover new behaviors and edge cases that may not have been covered in the learning phase.
- Validating: The system validates the application's behavior against the expected results and reports any deviations as potential bugs.
Discover How To automate Non-automatable Applications!
Challenges on the Path to Autonomous Testing
While autonomous testing holds great promise, there are also significant challenges that need to be overcome, such as:
- Complex workflows: Autonomous testing systems may struggle with complex workflows that require domain knowledge or human intuition.
- Non-deterministic behavior: Applications that have non-deterministic behavior, such as those that rely on external services or user input, can be difficult to test autonomously.
- Lack of standardization: There is currently no standardization around autonomous testing, which can make it difficult to compare different solutions or ensure interoperability.
- Cost: Developing and maintaining autonomous testing systems can be expensive, especially for smaller companies or those with legacy applications.
Myths about Autonomous Testing
Finally, let's debunk some common myths about autonomous testing.
Here are mine and Tobias’s top three:
- Myth 1: Autonomous testing will replace human testers. While autonomous testing can automate many tasks, it is unlikely to fully replace human testers, who will still be needed for strategic planning, exploratory testing, and edge case analysis.
- Myth 2: Autonomous testing is easy to implement. Implementing autonomous testing requires significant investment in tools, infrastructure, and expertise. It is not a quick or easy process.
- Myth 3: Autonomous testing is only for web applications. While web testing is a common use case, autonomous testing can be applied to any type of software, including mobile apps, desktop applications, and embedded systems.
Are You Ready for AI and Autonomous Testing?
To sum it all up, autonomous testing is a promising technology that has the potential to revolutionize software testing.
However, it is important to understand the key components, benefits, and challenges of autonomous testing, and to separate the hype from reality.
By taking a pragmatic approach and investing in the right tools and expertise, companies can start to realize the benefits of autonomous testing today, while also preparing for a future where AI and machine learning play an increasingly important role in software development and testing.
