2 Ways Test Automation Can Break Your Heart

Have you ever dated someone who left you for someone else? Without attempting to change the troublesome issues in your relationship they instead go for a quick fix — dating someone new and exciting, only to find out later that they’re still unhappy. It’s kind of like what my company does with frameworks and untestable applications.

1. Try to Automate Untestable Applications

I work for a company that uses an older application that wasn’t built with testability in mind. There are no unit tests for this application; and worse, there aren’t any hooks into the application itself that you can use to help automation. So when you spy on it it’s almost like a black box.

Almost none of the fields are recognizable. Automation has to rely, in large part, on x,y coordinates.

The current framework is a Frankenstein-like QTP/BPT approach. The automation engineers use all kinds of funky ways to interact with the application. As a result, the test failure rate is around 60%.

Refactoring can heal a broken automation heart

As you can imagine, no one is in love with this. Management goes on the war path every few years demanding that it be resolved, and each time we tell them the same thing: the actual application code needs to be refactored to make it more testable.

No one wants to touch the code. It’s old, and it would be time consuming. But it’s the right thing to do. So what do they always end up doing? Their solution is always to create a new, hot framework! WTF!?! Heartbreaking and anti agile test management.

Our current infatuation is with the White Framework with Specflow tech stack. I try to tell them that the issues we’ve seen with QTP are the same ones we’ll see with the new approach. But no one listens. (FYI nothing wrong with these tools it how you use them. For more info on the White Framework check out Automation Guild session on Fun Free Fast Automated GUI Test with White.)

Is it Testable?

It kind of reminds me of a great TestTalks interview quote from Kalayana Rao Konda:

“You may be able to build an app, but are you able to test it? Most teams fail to ask this question: ‘Is this going to be testable?’ during their requirements and test planning phase.”

Writing a new framework is fun and makes it appear as though something is getting done. Fixing culture, however — that’s another story. It’s like starting a new relationship without reflecting on what may have caused you to be unhappy in your last relationship.

Having an un-testable application also leads us to the second most popular way that automation can break your heart: focusing all your automation efforts on UI tests alone.

[tweet_box design=”default”]You may be able to build an app but are you able to test it? Ask ‘Is this going to be testable?’ during req phase[/tweet_box]

2. Focusing all your efforts on the UI

I'll admit it: UI testing is sexy. Everyone can see it, and become hypnotized by the way it manipulates their computer screens. I get it.

On some level, all of us begin falling in love based on looks. But if your relationship never moves past that superficial stage, you’re headed for heartbreak – much like teams that only create automation for end-to-end UI tests.

Not Everything Needs to be a UI Test

As I've said many times in other posts and podcasts, not everything needs to be a UI level test.

Here’s a great quote from Alan Page who works at Microsoft: “95% of the time, 95% of people write bad GUI automation. There are a lot of reasons for this, and a lot of times a GUI test is not the right type of test to write.”

Test at the Right Level

For example, if you're writing a lot of REST services and you have a page that consumes that RES service. I've seen teams that have twenty tests that go through the UI to test that REST service. WHY!? I understand you need to run tests at the UI level. But you could get that coverage by having just one user journey test that tests UI from the user perspective. The other nineteen tests can be at the API level.

The benefits would be that it will run quicker and faster. And API tests tend to be more reliable. Being able to run test fast to give developers feedback as soon as possible is key to being able to scale your devops and agile practices.

Go Headless

I began my automation career as a performance tester. We were able to ramp up hundreds and thousands of reliable users using LoadRunner. With LoadRunner, you simply capture the interaction between the browser and the application.

Because the tests were headless HTTP tests, I hardly ever had issues with those scripts. You do need GUI automation just like you need to have chemistry with someone to develop a romantic relationship. But I would focus more on unit, integration, and API-level tests.

Too Many Selenium Tests will Leave you Hearbroken

But don't take my word for it. For example, Jim Evans is one of the main contributors to the Selenium .NET bindings. No one can blame him for not liking UI tests. He brought up a great point during his TestTalks interview. When he hears people telling him they have thousands and thousands of Selenium tests, he gets a little worried, because he sees it as an indication that they’re trying to do too much with Selenium.

How to Avoid Test Automation Heartbreak

My advice would be to start focusing on fun, sexy, UI-based automation. But heed my warnings to avoid automation heartbreak.

Focus your testing efforts on ensuring your applications are testable and can be tested beyond the UI level. Then get ready to live happily ever after with your automation testing.

PS

Happy Valentines Day!