What is the Best Automation Testing Tool?

Two related questions I’m asked quite often are:

• What is the best automation testing tool?
• What framework is the best for automating web apps?

That’s why I was happy that someone actually asked these questions on the recent QASymphony webinar I co-hosted along with Paul Grizzaffi.

I thought the answers we gave really addressed this question, and wanted to share those insights with you.

No Best Automation Testing Tool

Both Paul and I are in agreement that there really isn't a “best” tool.

You're not looking for the best tool for Paul or the best tool for me. What you're really looking for is the tool that fits what you're doing.

Paul’s Advice on Selecting the Right Automation Tool

Paul recommends three aspects you need to address when choosing what the best automation testing tool is:

• What are your strategy and goals? What are you trying to get done?
• What problems are you trying to solve with automation?
• Who is your audience?

Do you have programmers on your team who know how to test? Are your testers or qa teams also programmers, or will it just be you who will actually be creating and running the automation framework?

Another question to consider is, what is your tech stack? You need to know this, because some tests tools are better at automating certain technologies.

For example — are you looking for a tool that can automate just a browser, or will you also be doing API testing? Some test tool technologies work well for driving a browser. Other tools can only handle API testing. There are others that work well for driving a thick client and a browser or mobile app testing , etc.

When you go for the non-open source vendor supply tools, that's where you’re typically going to have the ability to test all kinds of technologies — all from within one unified automation solution.

Future Proofing Automation

I actually think less about what the best software testing tools are and more about which tools are most appropriate for you, and will also build in some future proofing. This will allow your framework to evolve when something in your application changes.

Joe’s Advice for Selecting an Automation Tool

I think Paul really nailed it. I totally agree. I don't think there’s one tool that is “right” or  “best” automation test tool for every team.

For example, if the developers on your team are Java developers who want to stay in their Java ecosystem they’re most likely going to choose a tool or an automation solution that's going to be able to leverage Java. This will easily allow your developers to contribute to your automation without context switching between IDE’s and languages.

It's quite different than if you're a different team that maybe they're not really all hardcore developers but say that their hardcore Ruby developers are like Python, which is a scripting language that makes it easier to create automation without all the extra overhead.

You would choose a different solution. Or maybe you have a bunch of folks who have no development experience whatsoever and that’s going to dictate which tool you're going to use.

What is the Best Automation Testing Tool? It Depends

So when selecting the best automation testing tool for your team it really is a “depends” question as opposed to a “what technology to use?” question. You also need to think about scalability.

It needs to be scalable so that you're not forced into a decision that you made with the technology without realizing — hey you know I've got to look at the product roadmap and the product roadmap says a year from now we're going to integrate these other technologies.

Check Your Product Road Map

One of the things you should always consider when you're trying to decide what is the best automation testing tool to use is the product road map. And don’t forget to evaluate cost, including the maintenance of the tool you’ve selected.

You should also ensure that when you do create a framework that you develop it to be scalable – able to easily grow over time according to your needs.

Hiring Automation Engineers 

When considering the product road map and the future of your development project, keep in mind who you have employed.

What are the skills of your current work team? What is your goal for that team?

What is your is your hiring profile? Is your training profile going to change? Perhaps the tool that you are selecting isn't appropriate today, but you have a plan to make your team such that it is appropriate by date X or milestone Y? All these things need to be baked into to your tool selection.

How Hard is it to Find Testers with these Skills? 

Your decision will have a long-term impact on your organization. It can even affect your hiring process.

So keep this in mind when you’re selected a tech stack for your automation. Make sure it's tool or technology that you're going to be able to fill that role for.

I work for a company a very large company, and when we do it's very difficult to find people with basic skills, and even harder to find people with automation skills.

If you end up selecting a very obscure tool or technology that no knows how to use or is even familiar with, you may have issues finding the right folks to hire for that position.

Try Automation Tools Before You Buy/Build

I’m shocked by the amount of people who select tools they didn't try first. Lots of time and effort can be saved if we simply invested in a small proof of concept for each tool. Get a feel for how it's going work. Use it for a full sprint to see how it integrates with your agile development process.

Get your team's feedback before you make a decision.

Rather than being dogmatic and preaching from the top down, you instead should be getting buy-in from your teams. You definitely don’t want to force them to use a tool that they hate.

The best way to find the best automation testing tool for your team is to do that is do a small proof of concept and actually prove that the tool or technology stack you selected is actually something people are going to use, and is going to be supported by the rest of your teammates, because you don't want to be stuck being the only person that does automation because you selected such an obscure testing technology.

Automation Testing Resource

Have more question about how to get stated with automation testing? Check out my in-depth article on automation testing resources and best practices.