The Top 3 Automation Testing Tools (2023 Edition)

What are the top automation testing tools for 2023?

There are many great options for automation testing, especially vendor tools for enterprise companies that need to automate more than a browser. Still, in this post, I will only cover open source options for browser-based testing.

**Keep in mind this is not a ranking. Also, the “best” tool depends on your team's unique requirements and needs.

Get More Automation Testing Tips

But first, let's make sure we are all on the same page and understand what Automation Testing is.

What is Automation Testing

Automation testing refers to taking a repeatable manual process performed by a developer or tester and leveraging a tool to automate the process.

Automation helps you to accelerate running through numerous test scenarios to check that the results produced by specific actions or lines of code match the expected results. If not, it raises an exception to inform you that something went wrong.

The term automation testing can also be controversial (Not me; I'm old school :)), with many preferring “automated checking” or “automation in testing.”

When used correctly, automation can help with:

• Efficiency: Automation testing can significantly speed up the testing process as tests can be run in parallel and repeatedly without human intervention. This can save time, especially for regression, performance, and load tests.
• Accuracy: Manual testing can be prone to human error, whereas automated tests perform the same steps precisely every time they are executed. This ensures that the product is tested accurately and consistently.
• Coverage: Automated testing can easily execute thousands of complex test cases during every test run providing coverage that is impossible with manual tests. This helps to ensure that all features of the application are tested thoroughly.
• Cost-Effective: While the initial investment might be higher, automated testing can be more cost-effective in the long run. Once the test scripts are created, they can be used repeatedly at no additional cost and with less effort.
• Reusability: Test scripts are reusable and can be utilized through different phases of the software development lifecycle. The same test can be run on different software versions, ensuring backward compatibility.
• Reliability: Automated tests are reliable in delivering quick results. They can run at any time of the day, providing continuous feedback to the development team.

So, what browser-based automation tools should you know about?

Well, here are the current big three you need to know.

The first one, in no particular order, is…..

1. Selenium

No surprise here – let’s start with Selenium, the groundbreaking automation testing library that transformed the testing world.

Jason Huggins told me he came up with the idea for Selenium when working on a  time expense billing system at Thoughtworks in 2004.

Even though he didn't intend to create a testing tool, Selenium's innovative approach to automation testing took the industry by storm.

But there was a problem.

As time progressed, however, users experienced a gradual slowdown and a plethora of bugs, revealing the JavaScript sandbox as a potential roadblock for the project's future.

Undeterred, Simon Stewart stepped in around 2007 to save the day, embarking on a mission to develop the next-generation Selenium WebDriver. This cutting-edge evolution of Selenium breathed new life into the platform, ensuring its continued success as a staple in software testing.

Features and advantages of Selenium

Over the years, Selenium has fostered a large ecosystem of open-source projects that use Selenium WebDriver as a key functionality component. Selenium can be extended in various ways, and there are numerous drivers, bindings, plugins, and frameworks created and maintained by third parties that enhance its capabilities.

One major feature of WebDriver is that a standard, so it’s not going away anytime soon.

What's a W3C standard?

W3C standard

The World Wide Web Consortium (W3C) is an international community that develops open standards to ensure the long-term growth of the Web. When Selenium became a W3C standard, it gained international acceptance and recognition. This has several advantages:

1. Uniformity and Compatibility: As a W3C standard, Selenium must follow specific guidelines, ensuring uniformity across web platforms. This means developers can use Selenium with any browser that also adheres to the W3C standards, leading to better compatibility.
2. Improved Interoperability: Different systems and applications can work together more seamlessly with standard Selenium. This is because standards help ensure that the technologies used to build these systems are compatible.
3. Future-proofing: Standards frequently foresee future needs and changes. Therefore, as a standard, Selenium will likely maintain its relevance amidst the developing of new technologies and techniques.
4. Community Support: Standards are usually developed with the help of a large community. This means that any challenges encountered when using Selenium can be addressed by this community, leading to quicker resolutions.
5. Reliability and Trust: Standards are generally well-vetted, which means they are reliable. Companies can trust that a solution based on a standard has undergone rigorous testing and evaluation.

Being a standard is a huge benefit for developers and companies that rely on it for web application testing, as it guarantees reliability, compatibility, and future-proofing.

Other Pros and Cons of Selenium

Pros:

1. Language Support: Selenium supports multiple programming languages, including Java, C#, Python, Ruby, and JavaScript. This makes it a versatile choice for teams with diverse coding skills.
2. Browser Support: Selenium supports many browsers, including Chrome, Firefox, Safari, Internet Explorer, and Opera.
3. Large Community: Selenium has been around for a long time and has a large community of users. This means plenty of resources, tutorials, and solutions are available for common problems.
4. Standard: Selenium is based on open standards curated by the W3C and is entirely open source. That isn't always true of other frameworks you'll hear about, which often aren't based on standards and are only partially open source.

Cons:

1. Slow Execution: Selenium tests can be slower than newer tools like Cypress and Playwright.
2. Flaky Tests: Selenium tests can sometimes be flaky and inconsistent when written by inexperienced testers due, leading to false positives or negatives.
3. Complex Setup: Setting up Selenium can be complex, especially when it comes to configuring WebDriver for different browsers. (* One solution that solves this is the new Selenium Manager)

Keep in mind many automation testing solutions use Selenium under the covers. Selenium is just an API designed to automate browsers, whereas solutions that use it are more of a “battery-included” approach.

Many of the so-called “cons” of Selenium can be easily over using open-source frameworks like:

• SeleniumBase
• Selenide
• NightWatchjs
• WebdriverIo
• Serenity,

To name a few.

Using existing automation frameworks will save you lots of time and effort since they act almost like a wrapper on top of Selenium and include many ready-made features that you don’t have to design.

So, before you go down the path of developing a Selenium framework from scratch, make sure to check out existing Selenium-based frameworks first.

Also, Selenium BiDi is opening up even more functionality, which is a positive development.

What about if you are a developer and more into Javascript? The next option might be more appealing to you.

2. Cypress

Cypress is a developer-focused testing tool that helps developers write tests for the web.

Based on a podcast interview I did with one of the co-creators of Cypress back in 2017, I learned that The Cypress project was started around 2015 by Brian Mann, a developer who had been working in the field for ten years and had written over 10,000 tests.

He realized that existing tools for automated testing were unreliable, too slow to use with test-driven development (TDD), and the worst part of a developer's day.

He began experimenting with new technologies being introduced into browsers, like web sockets, to create an underlying architecture that was much less flaky, more powerful, and ran faster.

Cypress was developed with a focus on three things:

1. Setting up the tool in under 60 seconds
2. Creating a writing experience that is fast and reliable with a heavy focus on debugging ability
3. Focusing on the maintenance cycle

Cypress uses a completely different approach to testing than Selenium.

While Selenium WebDriver runs remotely outside the browser and executes remote commands into the browser, Cypress runs inside the browser.

When tests are run, Cypress boots the Cypress web application and then pulls in the user's application. This enables Cypress to synchronously get notified of everything inside the browser, providing native access to every single DOM element and every aspect of the application. This enables Cypress to make automation a lot more precise.

Cypress straddles both worlds, running inside the browser to do cool things but also expanding beyond the sandbox of JavaScript and beyond the limitations of JavaScript itself.

Cypress uses many of the same underlying automation APIs that the browsers expose but bypasses WebDriver and talks directly to the browsers the same way each driver does for each browser vendor. Cypress comes fully baked and fully packaged as a desktop application using electron and node.

Cypress is not a general automation tool but focuses on testing, ideally as developers build their applications.

Testing expert Gleb Bahmutov in the same podcast interview with Brian, mentioned that certain tradeoffs are made by using Cypress. While some specific situations may not work, it does much better than any other tool available in the situations it was built for.

Get More Automation Testing Tips

Unique features and advantages of Cypress

Cypress is a JavaScript-based end-to-end testing framework designed to simplify modern web application testing. It allows developers to create tests, debug them visually, and automatically run them in continuous integration builds.

Cypress provides a real-time view of end-to-end and component tests as you develop your applications. Its API is designed to be simple yet powerful, running as fast as your browser can render content. The installation process is straightforward, with no servers, drivers, or other dependencies to install or configure.

One of the key features of Cypress is its ability to run directly in the browser, allowing developers to debug failed tests using familiar in-browser developer tools. It also aims to eliminate flaky tests by interacting with your application in the same way as your users, helping you discover intermittent bugs before your users do.

Cypress can be integrated with any CI provider, making running tests in a CI environment easy. It also offers a cloud service, Cypress Cloud, which provides features like test parallelization, load balancing, and spec prioritization to optimize your test runs. The cloud service also offers visual tools for reviewing and debugging test failures, as well as in-depth analytics to monitor the health of your test suite.

This solution is loved by developers worldwide, with (last I checked) over 5 million weekly downloads and over 43,000 GitHub stars. It is praised for its ease of use, extensibility, and high-quality documentation.

Cypress Plugins

One main strength of Cypress I want to mention is its extensibility.

That means you can add things to the testing tool without waiting for it to add them.

Because of this, Cypress has an extensive library of plugins that extend its capabilities.

Some of my favorites that folks have shared with me on my podcast and webinars are cypress-axe which helps test your application for accessibility issues using axe-core, and cypress-audit, which allows you to run Google Lighthouse audits directly from your E2e test suites.

Other Pros and Cons of Cypress

Pros

• Cypress is designed to focus on front-end developers and provides a robust developer experience.
• It offers automatic waiting for elements, which can be useful in end-to-end testing where elements might not be immediately available.
• Cypress provides detailed error messages pointing to the exact line of code causing the issue, and it automatically records a video of the test run and takes a screenshot when the test fails.
• It has a concise syntax is easy to start with, making it a good choice for developers new to end-to-end testing.

Cons

• While Plugins are a plus, sometimes knowing that one is needed is hard. For example, handling iframes in Cypress requires a plugin. Not everyone knows this.
• Limited browser support
• No multiple-language support, so you’re stuck using JavaScript.

Maybe you are more of an SDET, and Selenium and Cypress aren't just doing it for you.

3. Playwright

The last of the big three is Playwright.

When I interviewed the previous program manager on the Playwright team at Microsoft Arjun Attam, in 2020, he described Playwright as:

“Playwright is a new cross-browser automation library for end-to-end testing. The goal with Playwright is to provide a single API developers and testers can use to automate their web applications. We developed it to make automation that is fast, reliable, and highly capable of the needs of the modern web.”

Playwright is the newest cross-browser Automation library for end-to-end testing of the three on this list, and it started off as a fork of Puppeteer in 2019 and is now maintained by Microsoft.

The philosophy behind it is different from the other tools.

It explicitly focuses on e2e testing, influencing the API available to developers and testers.

For example, actions in Playwright would auto-wait for the element to be ready.

Now, waiting for an element to be ready is a common pattern developers must explicitly write in their code. And Playwright does that automatically.

This makes it easier for you to write tests and makes your tests more reliable. And this is generally more important in the space of end-to-end testing.

Features and Advantages of Playwright

Playwright is a tool designed for end-to-end testing of modern web applications.

Ast the time of this writing, it supports all modern rendering engines, including Chromium, WebKit, and Firefox, and can be used across different platforms such as Windows, Linux, and macOS.

Playwright supports multiple programming languages, including TypeScript, JavaScript, Python, .NET, and Java.

One of the critical features of Playwright is its ability to emulate native mobile browsers like Google Chrome for Android and Mobile Safari.

It also provides robust features to prevent flaky tests, such as auto-wait functionality, web-first assertions, and tracing capabilities.

Playwright operates out-of-process, aligning with modern browser architectures and freeing it from the typical limitations of in-process test runners.

It supports testing scenarios that span multiple tabs, origins, and users and can interact with dynamic controls and produce trusted events.

To ensure complete isolation and fast execution, Playwright creates a new browser context for each test, equivalent to a new browser profile. This allows for full test isolation with minimal overhead. It also provides powerful tooling, including code generation by recording actions, an inspector for page inspection and selector generation, and a trace viewer for capturing all information necessary to investigate test failures.

Other Pros and Cons of Playwright

Pros

• Playwright supports a variety of selectors, including text selectors, CSS selectors, XPath, and special selectors for different front-end frameworks like React and Vue. This flexibility can be beneficial for developers working with different front-end technologies.
• It provides a selector picker in Visual Studio Code, which can help developers easily select elements on a page without manually inspecting the DOM structure.
• Playwright offers bindings in multiple programming languages (JavaScript, TypeScript, Python, Java, and C#), making it a versatile choice for developers with different programming backgrounds.
• It supports a headless mode for running tests, which can be useful for continuous integration environments.
• Multiple Browser Support: Playwright supports testing across multiple browsers, including Chrome, Firefox, and Safari.
• Mobile Emulation: Playwright supports mobile emulation, allowing you to test how your application behaves on mobile devices.
• Can easily leverage a Playwright test for performance testing, which is a big plus

Cons

• Smaller Community: Playwright is newer than Selenium and Cypress, so its community is smaller. This might mean fewer resources and solutions to common problems.
• Complex API: Some users find Playwright's API more complex than other tools, which could lead to a steeper learning curve.
• Playwright's syntax might be less intuitive for developers new to end-to-end testing, especially compared to Cypress's concise syntax.

Selenium Vs. Cypress Vs. Playwright Comparison

The Best Advice for Automating Your Tests

Finding it hard to keep up to date on the automation testing. Continuous learning is crucial to staying proficient in the field of testing.

If you want to maintain a strong foundation in timeless automation testing wisdom, check out my book Automation Awesomeness: 260 actionable affirmations to improve your QA and automation testing skills.

I share insights from interviews over nine years on the Test Guild podcasts in this book.

The objective is to inspire by sharing the perspectives of highly knowledgeable testers. Whether you’re a beginner or an experienced tester, you’ll find something in this book to help you improve your automation testing skills.

Get Your Copy of the Book Now

Which Automation Tool is Better?

Which tool for browser automation should you use?

As I always say, each team and situation is different. You need to try for yourself.

Some rules of thumb are if you’re a tester, try Selenium first. If you’re a front-end developer, try Cypress; if you’re an SDET, try Playwright.

I always recommend running a two-week proof-of-concept with any tool involving your whole team. Run it thru a real-world two-week sprint and see how it performs in your environment.