I recently posted this article on TechBeacon but thought I should share it on my blog as well. One of the most fundamental questions a test engineer asks before starting a new project is what tools they should use to help create their automated tests. Fortunately, Gartner issues a yearly report to help teams answer this very question: the Magic Quadrant for Software Test Automation report.
The Magic Quadrant for Software Test Automation report is divided into four main quadrants: Leaders, Challengers, Visionaries and Niche Players. You can read the full description of each quadrant in detail in the actual report, but the leaders in the market are determined by supporting all three of what Gartner considers the key use cases which are:
- The tools needs to support mobile applications
- Has to have responsive design
- Must support packed applications like ERP and SAP.
Vendors in this Year’s Magic Quadrant Report
In this year’s Magic Quadrant for Software Test Automation report, Gartner looked at the following twelve major vendors list (in no particular order ):
-
- Hewlett Packard Enterprise
- IBM
- Tricentis
- Worksoft
- Oracle
- SmartBear
- Ranonex
- Progress
- Automation Anywhere
- Bordland
- TestPlanet
- Original Software
I think it’s important to mention that Gartner does not endorse any of the vendors it reviews, and applies a non-biased policy to its survey research. So in theory, their rankings should be as non-partisan as possible.
Who are the “Leaders” in the Software Test Automation Magic Quadrant?
So which companies did Gartner select as the leaders in the test automation space based on their survey?
HPE
Of the twelve test tool vendors that were evaluated by Gartner, Hewlett Packard Enterprise (HPE) took the number one spot in the Leaders quadrant. Leaders are selected based on Gartner’s evaluation criteria of a company’s ability to execute and their completeness of vision. (Check out the Evaluation Criteria Definition section in the report to read more on this process.) But of all the companies evaluated this year, HPE matched this criterion best.
Some of the main reasons for HPE’s top spot was for what Gartner considers HPE’s breadth of coverage to deal with Agile practices and mobile application delivery, and awarded them major points for their broad network of service and technology provider partners.
Gartner also noted some cautions with HPE based on their survey feedback from customers, such as the lack of Linux support, and customers commenting that there needs to be a simplification of pricing and licensing.
Having been an HPE user for many years, I wholeheartedly agree with Gartner’s weakness assessment. The good news is that it seems that HPE has listened to their customers’ criticisms, and have begun to simplify in a number of areas — as seen with their new pricing and flexible consumption-licensing model for HP’s Lean Functional Testing (LeanFT).
IBM
The Number Two Leader spot went to IBM. (It’s worth noting that these rankings change every year; for example, in 2014 IBM had the top spot.)
Like most companies, IBM’s test tools portfolio has grown with the acquisition of tools like Rational and Green Hat. Much of the strength of their functional test tools comes from their support of many technologies and platforms including Windows, Mac and mobile platforms. One of their weaknesses is that some customers who were surveyed viewed IBM as slow on product innovation and support.
Tricentis
Tricentis is self-billed as “the continuous testing company,” which is in line with Gartner’s finding that one of their strengths was their strong efforts to support Agile testing and CI processes. Although it was also noted that they have a complex pricing model and larger testing projects, following Tricentis’ process can be tricky.
Worksoft
Worksoft is known for its ERP business end-to-end solutions. Ease of use and customer support were some of the strengths cited by the customers surveyed by Gartner. Customers also reported Worksoft’s integrations with other tools as a weakness.
That’s it for the leaders. The other companies fit into the remaining quadrants as follows:
The “Challengers” in the Software Test Automation Magic Quadrant
Challengers are seen a strong choices, but only support two of the defined use cases. This year there’s only one in this quadrant.
Oracle
Oracle’s application testing suite supports many things but is probably best known for their tight integrations with testing Oracle SOA and Packaged applications. Customers gave it high marks for its test management solutions, but also complained that the maintenance of its test scripts could sometimes be challenging.
The ” Visionaries” in the Software Test Automation Magic Quadrant
Visionaries, as the name suggests, are companies that can either breathe innovation into the market, or quickly support newer technology.
Bordland
Its been awhile since I’ve heard from Borland, so I was surprised to see them listed in this category, but then I noticed that they acquired Segue Software some years back. Old-school automation engineers will tell you that, back in the day, Silk Test was one of the more sophisticated, developer- friendly automation solutions available. So it’s no surprise that Gartner lists Borland’s strong support across many technologies and platforms as one of its strengths.
The main issue is that Bordland has grown by acquisition and mergers, so there is considerable confusion around their product messaging – which is probably another reason why I didn’t realize they were still operating in the test automation space.
TestPlant
TestPlant is one of the only test automation tools listed that has strong support for Apple’s platform. Because of its unique, image-based recognition abilities, it is known for its ability to test hard-to-automate applications; especially those with object recognition issues. Unfortunately, however, anyone who has done image-based, functional test automation knows how difficult these types of tests can be to maintain, and some customers noted that as an issue.
Original Software
Makers of the functional test tool TestDrive round out the visionary category. TestDrive can handle automation or browser- and legacy-based applications. Although I’m not a fan of “code-free” test automation solutions it did receive high marks from customers who reported that it is easy to use, and makes migrating from manual to automated testing simpler.
The “Niche Players” in the Software Test Automation Magic Quadrant
Niche players are usually those companies that either provide a unique testing solution to a specific use case, or are newcomers to the market.
SmartBear
I’ll be the first to admit that I have special fondness for Boston-based companies, of which SmartBear is one. To my knowledge, SmartBear has some of the more affordable test tools of all the vendors listed. They also have one of the more popular test tools for API testing: SoapUI. Gartner mentions that the strengths of Smartbear include their solid support for testing and their attractive price. Once again, however, a negative mentioned by Gartner is that Smartbear, like many other firms, has grown by acquisitions so the integration with their other tools can sometimes be a challenge. For example, as of right now I can’t make an API call to their ReadyAPi product from their TestComplete functional test tool like I can with, say, UFT GUI and UFT API in HPE’s Unified Functional Testing.
Ranorex
This is the first time that Ranorex has made an appearance in the yearly Magic Quadrant report. Besides having what I think are some of the coolest explainer-style videos, Ranorex supports a ton of technologies across all kinds of platforms — all from one tool — and Gartner lists that as one of their strengths. A caution that Gartner listed, however, which may not be an issue for all companies, is that they lack a full, end-to-end solution and focus mainly on functional test automation.
Progress
For those of you who may not be familiar with them, Progress recently acquired Telerik, which is also the home of the popular free debugging tool, Fiddler. I know a few test engineers that actually use Progress’s Test Studio as a front end for their Selenium test automation efforts. One of the strengths of Progress is its integration with Visual Studio and it’s supported languages. But Gartner believes you should be cautious when integrating with non-Microsoft technologies.
Automation Anywhere
I haven’t tried Automation Anywhere, but Gartner lists one of its strengths as a test automation solution for non-technical testers and teams. Users also report strong customer support for their products. Automation Anywhere does not support testing for packaged applications like SAP, and also lacks support for native mobile apps testing.
Where is Selenium?
That’s it for the quadrant test tool vendors evaluated in this year’s report.
Some of you who are reading this might be wondering why your favorite test automation tool (like Selenium) is not listed. The reason is that this report only focuses on tools that support what Gartner considers “key areas,” and the tools selected have at least two of the three specific use cases we talked about earlier in this article. If you don’t see one of your favorite tools listed, that might be the reason why.
My Key Takeaways
One interesting strategic planning assumption the report makes is that “By 2020, Selenium WebDriver will become the standard for functional test execution, and this will marginalize vendors that can't provide strong higher-level test functionality.”
I completely agree with this statement, and I think there is one main factor that will definitely make the adoption of Selenium WebDriver by vendors happen — a W3C standard.
W3C WebDriver Standard
There is currently a W3C WebDriver standard in draft form; once it is approved and becomes official, standard browser vendors will be responsible for creating their own browser-based implementation of Selenium WebDriver. Once that occurs, all the tool vendors will have to support Selenium because it’ll no longer be just their open source tool; it’s actually going to be a W3C standard.
That’s why Gartner’s statement really resonated with me. I believe most vendors are starting to better integrate with Selenium. For example, a little over a year ago SmartBear started adding integration in their Functional Test tool, TestComplete with Selenium. Today, from within the TestComplete IDE, you can include your Selenium WebDriver tests in a TestComplete test project, run them, and analyze the test results.
A more dramatic example is HPE’s recent announcement of LeanFT. Lean Functional Testing (LeanFT) combines the best of both the vendor-based and open-source worlds by morphing Selenium with some key functionality currently found in UFT. It essentially lets you create test automation scripts in the same IDE and languages as your Developers use, including Java and C#. Furthermore, you aren’t limited to just LeanFT SDKs for automation. You can combine both Selenium and LeanFT in the same project. I think we’ll be seeing more and more vendors taking this approach.
Open Innovation
Also the past two HP Discovers that I attended last year you could really sense that HPE has done a complete 180 and has now whole-heartedly started to embrace open source technologies. No longer is it an US v Them mentality that some HPE customers might have known in previous years.
I think more and more vendor will follow this open innovation model.
Since Selenium is essential just an API and not a test tool teams need to build out lots of functionality that might be taken for granted with vendor-based test tools. Accordingly, I can see vendors stepping up and seamlessly integrating with Selenium with enhanced functionality like reporting to make your automation efforts easier.
This open innovation will really help enterprise organizations, because they will no longer need to choose. They’ll instead be able to select the best tool/solution(s) for each of their projects.
Summary
Things change quickly in technology, and the Magic Quadrant for Software Test Automation report definitely helps me put my finger on the pulse of upcoming testing trends, and which tools support which technologies. It definitely helps me make better plans and choices when it comes to the tools and technologies I end up using on my projects.
I also feel that with more and more firms creating software faster and faster, and moving to a more DevOps/Continuous Deployment model, test automation is more critical than ever.
I don’t think you can succeed in today’s software development world without some sort of automation – just another of the many reasons it is an important report that is worth checking out every year.
Comments are closed.