Why Automated Tests Should Be Atomic (The Atomic Punk)

I recently spoke with Nikolay Advolodkin on my Test Guild Automation podcast about automation best practices. The importance of atomic automation tests came up, but I felt like we didn't talk enough about it.

Also, one of my childhood idols, Eddie Van Halen, passed away recently, and these concepts make me smile because it reminds me of his song, Atomic Punk.

But I digress.

Nikolay decided to create a post that takes deep dive into why it's so crucial for your automation tests to be as small as possible.

At this point, you might be saying “Enough, Joe! Please hand it over to Nikolay!”

Okay, okay. Here you go:

An Atomic Test Defined

This concept is critical for you to understand in order for any automation framework to be successful:

A single automated test should not mimic end-to-end automation.

Instead, your automated checks should form a single, irreducible unit. You should only be testing a single feature at a time.

If you have a software developer background, this is very similar to writing unit tests.

Your test is atomic if the following holds true:

• The test will only have one or two assertions at most. (Sometimes you need one assertion to make sure your application state is correct.)
• Your tests have very few UI interactions, and they're only on a maximum of two screens. In rare cases, an atomic test might navigate through three screens.

What are the benefits of going to the trouble of designing your automation testing like this?

There are many advantages of designing your tests to be as small as possible.

Fail Your Automated Checks Fast

One advantage is that writing atomic tests allows you to fail fast and early.

The feedback you get is speedy and focused.

It will take you no longer than one minute to check the state of a feature.

Next, atomic tests decrease flaky test behavior.

Less Flaky Automated Testing

Tests that you complete in two minutes or less are twice as likely to pass as tests lasting longer than two minutes.

Therefore, keep in mind that the longer a test takes to run, the more likely it is to fail.

Also, writing atomic tests reduces flakiness because it decreases the number of possible breaking points in that test.

You’ll see a decrease in false positives, which will, in turn, decrease the amount of time you spend on troubleshooting issues.

Here's an example:

1. Open UltimateQA.com home page
2. Assert that the page opened
3. Assert that each section on the page exists
4. Open Blog page
5. Search for an article
6. Assert that the article exists

Every single step is a chance for something to go wrong when using UI automation.

For example, a locator or the interaction mechanism may have changed, or your synchronization strategy may be broken.

Therefore, the more steps you add, the more likely your test will break and produce false positives.

Better Automated Testing Code Coverage

The third benefit of writing atomic tests is that they will not block other functionality from being tested if they fail.

In other words, the checks allow for better testing. For instance, refer to the test that's mentioned above.

If it fails on Step 3, you may never get to check if the Blog page or Search functionality works. (This is assuming that you don't have other tests to check this functionality.)

As a result of running a massive test, you will reduce your test coverage.

Finally, another great benefit of writing atomic unit tests is that they will run quicker when parallelized.

They're short and fast, significantly reducing your test suite's execution time.

For instance, in this refactoring, I got a 98% performance enhancement in test execution speed.

In the scenario above, I had a suite of 18 end-to-end tests that were NOT atomic and were not running in parallel.

Then, I broke down my tests into 180 tiny atomic tests while maintaining the same code coverage.

Next, I ran them in parallel and decreased the average execution time of the test case to 1.76s, all the way from 86s!

How to Break up End-to-End UI Tests to be Atomic?

By now, you probably agree that atomic tests are a good idea.

However, you may be wondering how to break up your large, end-to-end tests.

Trust me – you're not the only one struggling with this situation. I encounter clients daily who have the same issue.

Furthermore, I’d like to provide a simple answer to this issue, but I'm unable to.

For most individuals, this challenge is one of technology and culture.

The good news is that I will provide a step-by-step guide to help you create atomic test runs.

While it may not be easy, when you do achieve it, it will be SO sweet!

Here is a simple scenario:

1. Open Amazon.com
2. Assert that the page opens
3. Search for an item
4. Assert that item is found
5. Add item to cart
6. Assert that item is added
7. Checkout
8. Assert that checkout is complete

The first problem is that many automation engineers assume you must do an entire, end-to-end flow for this automated test flow.

For example, you must complete Step 1 before Step 2, and so on.

The reason being, how else can one get to the checkout process without having an item in the cart?

Inject Data to Populate UI State

The automated testing best practices approach is to inject data to populate the application's state before any UI interactions.

This will significantly help your testing process. For instance:

You can control the state of an app via several options:

• Using something like a RESTful API to set the application into a specific state
• Using JavaScript
• Injecting data into the DB to set the application in a certain state
• Using cookies

If you can inject data between the application's seams, you can isolate each step and test it on its own.

Some options to consider:

• Send a web request that will generate a new test user
• Send a web request to populate an item in your Amazon Cart
• Use Selenium to open the browser to the Cart page
• Perform the checkout using web automation
• Clean up all test data after

Using a web page API is much more robust and faster than using a GUI with AUI Automation for each test step.

For instance, a web request can execute in roughly 100 milliseconds.

This means that steps 1, 2, and 5 can each take less than one second to complete.

The only other step you will need to complete is to finish the checkout process with Selenium (the only part that you actually want to test).

How to Control App State using JavaScript

One of the most common impediments to atomic testing is the login screen.

The upside is that most apps have one.

So how do you remove this from your test so that your test can be atomic?

Here's one example:

Page with a login screen

1. Open the web app using a GUI testing tool
2. Execute JavaScript

Congratulations, we are now logged in!

Now, use your GUI automation testing tool to perform the single operation you want to test.

Here's how a full atomic test would look:

Notice how the test only has:

• overviewPage.Open() – one GUI action for setting state
• overviewPage.Cart.SetCartState() – the method to set cart state with JS
• one more GUI action for overviewPage.FinishCheckout() – this is the feature that we want to test
• and one assertion

Can't inject data for testing with your web application?

What if you can't inject data for testing?

The world isn't perfect, and you will face challenges in your role.

Many people aren't lucky enough to have applications that are developed with testability in mind.

Be glad to know you have two useful options if you can't inject data for testing.

For instance, you can work with software developers to make the application more testable.

A sign of poor development practices is not being able to test your application easily. Communication and sharing of ideas are all part of the job.

Communication Critical for Longterm Test Success

Be prepared to leave your cube and communicate across silos to break down communication barriers.

You’ll need to communicate between teams and work together to create a stable product. The whole team fails when a product fails, not just a specific group.

You might be thinking, “Yes, but it's not easy.”

I agree. I worked at one company where it took me a year to simply integrate developers, automation tools, and manual QA into a single CI pipeline.

It was a weekly grind to get and keep everyone caring about the outcome of the automation suite.

However, in the end, our team was more robust and more agile than ever for it.

Trust me, it's doable, and most developers are happy to help.

The main point is that you must be willing to break down these barriers if you want to succeed.

Then, of course, there's option two, but you may not want to hear it.

Don't automate it if your application is not automation-friendly.

If you can't work with the developers because you're unwilling, or because the company culture doesn't allow you to, then don't automate something that won't provide value (even if your manager has asked you to automate it).

We are automation engineers. We are the professionals. We must decide what to automate and not to automate based on our understanding of application requirements.

We are hired because of our technical expertise and abilities to say what is possible, what is not possible, and what will help the project succeed.