After blogging for over ten years, I realize that I never defined what automation testing is.
So, let’s go over the basics to set you on your way to creating what I like to call automation awesomeness.
Definition for What is Automation Testing
Automation testing refers to taking a repeatable manual process performed by a developer or tester and leveraging a tool to automate the process.
Automation helps you to accelerate running through numerous test scenarios to check that the results produced by specific actions or lines of code match expected results. If not, it raises an exception to let you know that something went wrong.
This approach is critical because it speeds up the time to test since you are using a scripted process that evaluates if the software developed actually meets your requirements.
We call this automation testing because you can easily use your computer to rapidly run through a ton of test cases/test scenarios in minutes.
Automation can save you a lot of time since they don’t have to perform these repetitive tasks repeatedly after changing an application under test.
In a nutshell, automation testing is a technique used to improve the execution speed of verification/checks or any other repeatable tasks in the software development lifecycle.
The term “‘automation testing” can also be controversial with many folks preferring to use the term automated checking or automation in testing.
In any case, before we take a look at automated testing, let's touch on some issues with manual testing.
Pitfalls of Manual Testing
There are a few reasons why manual testing may be problematic:
- It uses a lot of resources
- It’s time-consuming
- It sometimes lacks proper coverage
- Due to their repetitiveness, testers may get bored and miss steps when executing manually, leading to possible inconsistencies.
Like automated testing “manual testing” can also be a controversial term. In fact, Michael Bolton has gone as far as to say that manual testing does not exist.
Automation testing should be used to help with your testing efforts.
But does automation replace all your tests?
Here's the deal:
Scope of Automation: Does Automation Testing Replace Testers?
Some people assume automation means replacing human testers. In actual fact, however, it’s the opposite. Automated scripts are great for running tests precisely and quickly but they in no way replace human testers.
Automation is also great for running the same steps over and over again, but they don’t actually think.
Although we can agree that automation testing does not replace other application testing activities, with today’s software development environment and continuous integration practices it is critical. With the increased speed at which we develop software, we need automation testing.
Another reason is that Agile and DevOps practices demand more automation. Practices like continuous integration and delivery require automated scenarios that run quickly and reliably. I’d actually go so far as to say that in today’s modern development environment, we can’t succeed without automation.
So what are some reasons for using automated tests in quality assurance?
For more info on the difference between an automation testing engineer and a tester along with typical automation testing salary info check out SDET vs. Tester: What's the Difference?
Benefits of Automation Testing
There are many benefits of automation for both testers and developers.
For developers, when we talk about automated tests, this would include unit testing, component testing, and integration testing.
Although most developers that work on modern software development have embraced writing unit tests driven development, this is not always the case on legacy systems.
Some programmers are afraid to modify an existing code due to unattended consequences of breaking something with their code change.
But given a choice, it is always safer to have tests around any changes that you make. When you change code, you can introduce errors; But when you cover your code with tests before you change it, you're more likely to catch any mistakes that you may have submitted with your change.
So one of the main benefits of any automated test is that it acts as a safety net. When you run tests for every new build of your application before checking in your code, you can have some level of confidence in knowing that you have not broken anything.
During regression testing or smoke testing, a manual tester will take an existing test case procedure and execute it step by step against a web app or API. This can be time-consuming since it is a manual process done by hand.
Thus, to save time, many companies try to take their manual regression tests and convert them into automated test cases. An automated test tool then executes the test steps automatically without human intervention.
This highlights two other benefits of test automation: 1.) it frees up the tester's time to focus their energy on high-value testing activities that can't be automated, like exploratory testing. And 2.) it avoids missed steps that can occur when performing a test manually.
With the increased speed at which we develop software, we need automation testing. As more companies move toward Agile and DevOps, automation is more important than ever before.
Practices like continuous integration and delivery require tests that run quickly and reliably. Lots of manual verifications will stop your ability to achieve velocity with your software development.
I’d go so far as to say that in today’s modern development environment, we couldn’t succeed without automation.
Although the main reason teams try to create automation is to save the company both time and money, it’s also important to give developers quick feedback so that when they check in code, they are alerted as soon as possible that the change they checked in broke something.
The theory is that an automation script will save the company both time and money. But it seems that many folks fail to factor in the amount of time and money it takes to maintain automated test suites.
Some other benefits of automated testing are:
- Verify newer versions of the software
- Free testers up to focus on more exploratory-type testing
- Automated scripts are more repeatable
- Data population
- Accurate benchmarking
- Less false failure due to human error
- Greater test coverage
- Quicker release of the software
- Get fast feedback to your developer on failing checked in software
- Saves time
- Ability to leverage programming capabilities
- Run for every new build of your application to act as a safety net
So are there any downsides to creating automation?
What's the real story?
Disadvantages of Test Automation
Since automated testing relies on programming languages for their creation, automation becomes a full-blown development effort.
As a matter of fact, what you are doing is developing a piece of software to test another piece of software. Treat your automated code just like your development code. Follow the same processes and best practices you would use for any other software development project
Automation testing is difficult and complicated, just like most other development software projects. It also presents many of the same issues other software programs do. So the same best practices for developing software also apply to automation as well.
Automation Testing is a Development Activity
Since automated suites usually rely on programming languages for their creation, automation becomes a full-blown development effort.
What you are doing is developing a piece of software to test another piece of software.
Automation testing is difficult and complicated, just like most other development software projects. It also presents many of the same issues other software programs do. Treating your automated code just like your development code is essential.
You must follow the same processes and best practices you would use for any other software development project if you are going to succeed.
If you ignore this advice you automation project will fail.
Other Automated Testing Problem Areas
Some other automation testing pitfalls are:
- Setting unrealistic goals
- Believing that automation will find more new defects
- A false sense of security
- Underestimating the amount of time it takes to maintain your web app automation
- The creation of large, end-to-end tests should be avoided. Scripts should be atomic so that when they fail, you know why
- Focusing on UI automation types of testing only
- Not having a controlled test environment
- Ignoring failing tests
- Not having a test data strategy in place
- Not reusing automation code
- Developers not making their code automatable
- Not using proper synchronization in your tests
- Not making your automated test readable
- Creating automated tests that add no value
- Hard coding test data
- Not using a whole team’s collaborative automation efforts
- Expecting automation testing to replace manual testers
- Hard to discover all the technology needed to create an end-to-end automated solution
Because of the above reasons, teams often claim that automation testing “doesn’t work.”
Another common question I’m frequently asked is, “Which tests should be automated?”
What Test Cases Should be Automated?
You shouldn't try to automate everything. In fact, not everything is automatable. When planning what test cases to automate here are some things to look for:
- Tests that are deterministic.
- Steps that don't need human interaction
- Tests that need to run more than once
- Any manual process that will save engineers time (not necessarily an official “testing” process)
- Test that focuses on the money areas of your application
- Test that focuses on the risk areas of your application
- Unit tests
- Tests that need to run against different data sets
- Tests that are hard to test manually.
- Focus on critical paths of your application
- Tests that need to run against multiple builds and browsers
- Tests used for load/stress testing
The more repetitive the execution is, the better candidate a test is for automation testing. However, every situation is different.
Ultimately, you should consider using automation for any activity that saves your team time. It doesn't have to be a pure testing activity; you can leverage automation to help reduce the length of time-consuming activity throughout the entire development life cycle.
Types of Automation Tests
Remember don't just focus on functional tests for your automation strategy.
Here are some other areas that could benefit from automation:
- User Experience
- Component testing
- Accessibility testing
- Exploratory testing using smart AI crawlers
- Unit test
- Integration test
- Environment Setups
At this point, some of you may be asking, “What is the ROI of test automation?”
Automation Testing ROI (What is the Cost of Test Automation)
Determining the ROI of your automation testing efforts can be tricky. Here is a common calculation some folks use to get a rough estimate of their test automation costs. This can also help you decide whether a test case is even worth automating as opposed to running testing it manually.
Automation Cost = how much the tools cost + how much the labor cost to create an automated test + how much it cost to maintain the automated tests
Consequently, if your automation cost calculation is lower than the manual execution cost of the test, it’s an indicator that automation is a good choice.
Moreover, ROI quickly adds up with each re-run of your automated test suite.
Because it’s critical that you get a good return on your test automation investment, there are some things you shouldn’t automate.
Why does this matter?
What Not to Automate
There are exceptions to everything, of course, but in general, you may not want to automate the following test case scenarios:
- One time test
- Ad hoc based testing
- Test that doesn't have predictable results
- Usability testing
- Application not developed to be testable
In addition to what to automate, another element of a successful automation project is having an automation framework.
What is an Automation Framework?
An automation framework is a common set of tools, guidelines, and principles for your tests. Having an automation test tool framework helps to minimize test script maintenance. In my opinion, common pieces and best practices of a framework are:
- Set your manager's and team's expectations
- Break your automation framework into abstraction layers
- Use proper synchronization methods
- Determine a strategy for training and retraining your framework users
- Use version control
- Look for existing libraries/tools before inventing your own
- Do code reviews on all automated tests
- Include reporting and logging that make debugging easy
- Follow a naming convention
- All elements should have unique IDs
- Avoid coordinate-based automation
- Create reusable methods
- Create reusable utilities
- Refactor code often
- Use Page Objects
- Make tests readable. (Your tests should read like English)
- Avoid code duplication
- Have a test data management strategy
- Handle running test in parallel
- Support mocking and stubbing
- Include automation on your sprint teams definition of done (DOD)
- Remember that automation is a collaborative, “whole team” effort
- Separate your tests from your framework
That's not all…
Successful Test Automation Process
It’s a given that your applications are going to change over time. And since you know change is going to happen, you should start off right from the beginning using best practices or design patterns, like using page objects, doing so will make your automation more repeatable and maintainable.
Automation Testing Process:
Here are six steps to follow in an Automation Process
First, prepare – understand the functional testing objectives. Understand what test data is needed. Know what needs to be verified.
Turn the requirements into an automated solution. Know what the start and end conditions are for each test. Tests should be completely independent of other tests. Add proper assertion checks to ensure your application is behaving according to your specifications. Each test should have a particular purpose.
Execute your tests should be reliable. Run each test at least three times in a row before checking in code.
verify that the automated test is doing what you expect it to do. Have manual tests verify that it is working as expected. Remember — if it’s not asserted, it’s not checked.
Be sure that everyone on your team is aware of the results. Flaky tests should be fixed ASAP, or you'll risk your team ignoring your test results.
if you notice a flaky test, refactor it to make it more reliable. Most importantly, deletes any tests that are not reliable and haven’t been fixed within a given time frame.
With Automation Testing – Test Size Matters
Because tests need to run quickly, test size matters. At this point, many people visualize a traditional test pyramid, which has unit tests as its base. Integration tests are in the middle, and GUI tests are at the top.
But I think more in terms of test size. By test size, I'm referring to tests that are faster than others. While I understand the need to run UI tests if you have to create one, make it as fast as possible.
- Don't do all UI end-to-end journeys.
In addition, when an automated test fails you need to know why. Furthermore, you should endeavor to get feedback to your developers as quickly as possible, and the best way to do that is with a fast, well-named test.
- Unit test, integration test tend to be faster.
If you're into Star Wars, you notice that Yoda in the image above is the smallest test size, and they tend to be more powerful too. As you move up the pyramid, UI tests can be evil and can drive you crazy, so although you need them, just be careful.
Automation Test Engineer Salary
You might have seen recent job postings in areas like Silicon Valley looking with the title of SDET. Most of these positions require someone that knows how to create test automation in a programmatic way. The good news is that many of these jobs offer higher salaries than for a standard tester.
How much more? On average I've seen the difference to be over $32,000! Check out my post on SDET vs. Tester: What’s the Difference? (The answer may surprise you!) for more detail info.
Test Tool Selection – How To Pick
There is no “correct” test tool for automation testing. Ultimately, it all depends on your team’s unique needs and skill set. Also, you sometimes need a combination of different tools to get the coverage your application requires.
Due to this, there are a few things to look for when trying to select a test tool:
- Look at the product roadmap and make sure the tools you select will handle future features and technologies.
- Evaluate the cost, including maintenance.
- Use a tool that a leverages the same tools and languages your developers use.
- Don't just assume a tool will work for you. Create small POC for each tool and get team feedback before committing to anything.
- Is the tool extensible?
- How easy is it to use and get started?
- Does it provide reporting and debugging capabilities?
- Does it recognize all the objects in your application?
- Can it integrate with other tools like version control, test management tools, continuous integration tools?
- Find out if the tool has an active user base.
- Select tools that other companies are using.
- How much training will it take to get your teams up to speed with the tool?
- Finally, determine how easy is it to hire folks that have the skills needed to create your automated tests
Automation Testing Frameworks & Tools List
Open Source Tools for Automation
Selenium – has arguably become the de facto test tool standard for browser-based testing for automation engineers. Please remember: you cannot use selenium for non-browser applications; not everything can be an automation testing Selenium script.
The first thing folks usually think of when they hear automation testing is automation testing selenium. Remember Selenium is just for browser automation. Also, it's not a tool its a library. Check out some of the other automation tools not named Selenium
Automation for apps. Appium seems to be the winner in the mobile testing space so far.
An open-source Ruby library for automating tests. Watir interacts with a browser the same way people do: clicking links, filling out forms and validating text.
Sikuli – What's cool about SikuliX is that it allows you to automate anything you see on your screen using image-based testing.
WinAppDriver – Windows Application Driver is a service to support UI Test Automation of Windows Applications.
AutoIt – AutoIt v3 is a freeware BASIC-like scripting language designed for automating the Windows GUI and general scripting. Many teams integrate AutoIT with Selenium to work around non-browser windows that appear in an automated test workflow.
Visual Validation Tools – The Top 21 FREE Visual Validation Tools for Testers
Serenity – One of my favorite automation frameworks around. Serenity is a great open-source tool because it acts as a wrapper over Selenium and Behavior Driven Development (BDD) tools like jBehave and Cucumber JVM. That means there's a lot of built-in functionality available to you in Serenity that takes care of many things you would normally have to code from scratch if you had to create your own BDD framework. What Serenity is really awesome at is creating unbelievable reports. Out-of-the-box Serenity creates living documentation that can be used not only to view your Selenium BDD test results but also as documentation for your application.
Sahi – The first thing you need to know is that Sahi comes in two flavors: open source and a pro version. Sahi Pro is the enterprise version of the open-source project. It includes lots of features coveted by larger organizations.
Robot Framework – If you want to use Python for your test automation efforts, you can’t go wrong using the Robot Framework. The Robot Framework is a mature solution that was created for testers and uses a keyword-driven approach to make tests readable and easy to create. It also has many test libraries and other tools you can use
Galen – If your automation efforts are focused on User Experience Design (UX)/Layout testing, Galen Framework might be a perfect fit for your needs.
Vendor-based Automation Test tools
Applitools – Actually Applitools integrates with both Vendor and Open-source solutions. If anyone has tried doing any sort of visual testing using tools like Eggplant and UFT Insight, you know how hard it is to make these types of tests reliable. Sometimes the tests are so fragile you can only run them on the same machine they were developed on to avoid flaky tests. There are many reasons for this, but it's mostly due to pixels being slightly off from one browser or OS. Applitools is different in that it was developed from the ground up for visual validation and its sophisticated algorithm was designed to handle many pixel issues that most other image-based testing tools have a hard time handling. Applitools allows you to find and automatically detect all the visual bugs to validate the visual correctness of your application.
Essentially combines the best of both the vendor-based and open-source worlds by morphing Selenium with some key functionality currently found in UFT (QTP).
IBM Rational Functional Tester
Like most companies, IBM’s test portfolio has grown with the acquisition of tools like Rational and Green Hat. It appears that much of the strength of its functional test tools comes from its support of numerous technologies including Windows, Mac, and mobile platforms.
Tricentis – Self-billed as “the continuous testing company,” which is in line with Gartner’s finding that one of its strengths is its extensive efforts to support Agile testing and continuous improvement processes.
Worksoft is well known for its ERP business end-to-end solutions.
One of the few test automation tools listed that has strong support for Apple’s platform. In fact, because of its unique, image-based recognition approach, it has the ability to test hard-to-automate applications — especially those with object recognition issues. Unfortunately, anyone who has done image-based, functional test automation knows how difficult these types of tests can be to maintain, and some customers noted that as an issue.
Supports a ton of technologies across all kinds of platforms — all from one tool. Noteworthy, however, is that it lacks a full, end-to-end solution and focuses mainly on functional test automation.
For those of you who may not be familiar with this company, Progress recently acquired Telerik, which is the home of the popular free debugging tool Fiddler. Also, I know a few test engineers who actually use Progress’s Test Studio as a front end for their Selenium test automation efforts. Strengths of Progress are its integration with Visual Studio and its supported languages.
Does not support testing for packaged applications like SAP. Also lacks support for native mobile apps testing.
Katalon Studio is an open-source solution a paid vendor tool that has functionality like Record and Playback, reporting, Object Spy, Object Map, and more. You can use it to create automated tests for Web, API, and mobile apps. It even runs on both Windows and Mac.
API Automation Test Tools
Open source API tools
Rest-Assured – Rest-Assured is an open-source Java Domain-specific language (DSL) that makes testing REST service simple. It simplifies things by eliminating the need to use boiler-plate code to test and validate complex responses. It also supports XML and JSON Request/Responses.
RestSharp – Simple REST and HTTP API Client for .NET
Postman – Postman is a rest client that started off as a Chrome browser plugin but recently came out with native versions for both Mac and Windows.
SoapUI – is the world leading Open Source Functional Testing tool for API Testing. It supports multiple protocols such as SOAP, REST, HTTP, JMS, AMF
Fiddler – Fiddler is a tool that allows you to monitor, manipulate and reuse HTTP requests. Fiddler does many things that allow you to debug website issues, and with one of its many extensions you can accomplish even more. Check out my article on how to get started with Fiddler
Karate – Since Karate is built on top of Cucumber-JVM, you can run tests and generate reports like any standard Java project. But instead of Java – you write tests in a language designed to make dealing with HTTP, JSON or XML – simple.
Vendor API Tools
SoapUI Pro – Since the free version is open-source, you can actually gain access to the full source code and modify as needed. The pro version is more user-friendly, and has additional functionality including a form editor, an assertion wizard for xpath, and SQL query builder.
UFT API – In previous releases HP had separate products for functional testing. QuickTest Professional (QTP) was used for testing GUI applications, and Service Test was for testing non-GUI technologies. HP's latest test tool release — Unified functional Testing (UFT) — combines both products and features a frontend that merges the separate tools into one common user interface.
Test Execution Report Tools
Allure – an open-source framework designed to create test execution reports clear to everyone in the team.
Run Your Automated Test in the Cloud or On Mobile Devices
Here are some vendors that allow you to save a ton of time by running your test in the cloud and on multiple OS, devices, configurations. Get rid of the headache of having to maintain your own in-house lab/grid.
Automation Test Management Tools
Zephyr – Manage all aspects of software quality; integrate with JIRA and various test tools, foster collaboration and gain real-time visibility.
QaSymphony – Has a platform called qTest for Software testing and QA tools built for Agile
Automation Testing Courses
Finally, here's a list of my favorite automation testing courses:
Automated Web Testing with Selenium – John Sonmez
Creating an Automated Testing Framework with Selenium – John Sonmez
Quick Guide to API Testing with HP's Unified Functional Testing – Joe Colantonio
Test Automation with CodedUI
Selenium WebDriver Basics with Java – Alan Richardson
Complete Selenium Webdriver with C# building a Framework – Nikolay Advolodkin
The Java Selenium Guidebook – Dave Haeffner
Test Automation Conferences
AutomationGuild – Automation Guild is the first-ever event of its kind, 100% online conference that is dedicated to helping YOU perfect the craft of creating automation awesomeness and accelerate your automation career. If you missed the LIVE Automation Guild event? No worries! Due to demand, I decided to keep registration open. So you can still get all pre-recorded sessions and recorded Q&A now!
PerfGuild – the first ever online conference dedicated 100% to performance testing.
SeleniumConf – SeConf brings together Selenium developers & enthusiasts from around the world to share ideas, socialize, and work together on advancing the present and future success of the project.
STPCON – The Software Test Professionals Conference is the leading event where test leadership, management and strategy converge