I’ve been involved with test automation for over 15 years, and I’m amazed that to this day there are misconceptions about test automation that most managers are not even aware of.Lets face it your boss doesn’t know test automation, but sometime they act like they do. This needs to stop.
Here are the top five things your boss probably doesn’t understand about test automation, and why you need to educate him or her ASAP:
1. Changing test tools is not going to solve your automation reliability issues.
It never fails. When it comes to crunch time for a release and our automation tests are acting unreliably, I always hear grumblings that the issue is with our test tool.
This was one of the main reasons a team I worked with moved from QTP to Selenium. For some reason, the boss thought that switching would improve their automation efforts and automation tests would start running reliably without much effort.
Recently this same team started blaming the test tools again for their test being unreliable and the manager wanted to look into changing tools. His suggestion? Wait for it – UFT! No joke.
It’s like saying the application you created in C# sucks, so rewriting it in Java will solve all your problems. Totally stupid. The problem is not your test tools – it’s your teams.
Changing tools is not going to solve any team dysfunctions you might be dealing with that are causing your automation efforts to be either ignored or neglected.
2. Test Automation is not just a QA activity.
This one will definitely get you into trouble if you don’t set your boss straight right away. When a team is involved in creating an automation framework, it becomes a software asset — just like any other development asset that needs to be maintained. Automation should also possess the same standard of code quality and coding standards. Accordingly, you should have your developers develop your automation framework — not a QA resource.
I don’t think many managers understand this. They believe that because it’s testing that it’s just a QA activity, so testers should be the ones who build the automation framework. And if you have a lot of QA people that have never written any code and you kind of throw them into that environment, you should expect that you're not going to get a quality framework. You are basically asking them to develop an application that tests another application. Since it is software development, why would you have a QA resource and not a developer to handle it?
3. Not everything should be a UI test
If your boss wants everything to be a UI test, watch out. If you're currently running hundreds or thousands of automated UI tests, it’s probably an indication that you're trying to do too much with Selenium or QTP. Based on my conversation with my TestTalks guests, I know of a number of companies that do have very large Selenium/UFT test suites that are actively trying to pair that number down and move a lot of that testing into unit and integration tests that don't require the full browser stack.
If you are in this situation, you need to educate your boss as to why a lot of those tests need to be gotten rid of. You should also take a long, hard look at why your manager believes that everything should be tested through the UI layer.
A helpful resource that offers info about a good ratio of end-to-end test vs. smaller integration and unit-type tests is How Google Tests Software.
4. Test automation does not replace manual testing
No matter how good your test automation scripts are, they will never replace someone that does some sort of manual/exploratory testing against your application. It’s always going to be a combination of very high-end intellectual manual testing and test automation. It’s a combination.
Simply having automation tools, or a large percentage of your tests automated does not equal quality. In my experience, many managers think that since they’ve made a large investment in either developing an automation framework or purchasing some test tools that it is a cure-all for their quality issues.
That is never going to be the case. Automation can only test what you tell it to. But the intellectual components of testing — really understanding your application’s design, and exploring the application with a tester’s mindset — those are the real intellectual elements. Tools will never be able to do that for you.
I’ve written another post that talks about this concept in more detail, called “Can automation do actual testing?” where I interview Richard Bradshaw, AKA Friendly Tester about his automation in testing concept.
5. Faster is not necessarily better.
Just because you have lots of automation doesn’t mean your application quality will be better. Just the other day I was talking with the manager of another team, and he showed me an executive dashboard that showed his project has 98% of all tests automated. He was really proud of that, but it scared the heck out of me. I’m guessing these tests either suck at actually testing important things, or aren’t really doing anything. To me this metric points out negligence, not quality.
That conversation actually reminded me of a one I had with Gojko Adzic, and some of the ideas he covers in his book Fifty Quick Ideas to Improve Your Tests. Gojko mentioned that what he often sees, especially working in teams that are stuck with a ton of automated test they can't maintain, is that they've done a “fast food” equivalent of testing; meaning they've written a bunch of things that were really easy to write and automate, but are ineffective in uncovering issues with their application.
So, sure — in theory, they may have 98% of their tests automated and can run their test suites faster than they could before, but those factors alone don’t make them better. As they accumulate, they start suffering more and more, because when you have tests that are hurting, automating them only makes them hurt more often.
Changing Your Bosses Test Automation Mindset
The point of all this is not to ridicule managers. In fact, the problem is not your boss – ultimately, it’s you that’s the problem (me included) because it shows that we as testers haven’t done a good enough job of educating management as to what the benefits of test automation are (and are not).
Yes – your boss doesn’t know test automation, but we need to stand up and correct them when they make ridiculous demands, and explain to them why it doesn’t make sense. Unfortunately, we often take the easy route and humor them by following their ill-advised wishes, or by gaming the system to prop up metrics with bogus numbers that give management a false sense of security.
Instead, we should focus on and care about what really matters: the quality of our applications, and ultimately, our application users.
Hahaha, Joe, your first misconception is too true. In fact, I can’t believe that this manager went from QTP to Selenium to UFT hahahaha. We have the same problem where I work. Where people believe that changing a tool will solve their automation problems. When they need to look at how they are designing their automation tests. I can write a very solid test in QTP, UFT, or selenium. Yes, Selenium makes writing tests much easier with their well thought out architecture. But it doesn’t make my tests better just because they’re in Selenium. But C# is definitely better than Java! All the applications made in C# are the best Hahaha.
Thanks Joe, good and accurate article. Really like point 1 :)
Hi Joe,
Great article, one small suggestion though. If all titles are conclusions about misconceptions, imho the second title should also be negative (.. is not just a QA activity).
Hi Ronald. Great catch! I made the edit – Thanks!!
HI Chris – its funny (or sad) that I’m not the only one that has experienced #1 :)
HA – I know Nikolay – I didn’t know if I should laugh or scream when the manager told me this.
I agree with almost everything. Number 2 I sort of agree with but have a different take on it. You absolutely need a competent developer to help architect and design the automation framework, and help build out the product abstraction layer (page objects in UI automation), but there are quite a few people in QA roles who I know are competent enough developers to do that job.
I have actually never been on a team where developers wrote automated acceptance tests and it was successful. I have also never seen a team full of manual testers with no development experience be successful either. What I have seen that has been successful are teams with a senior, competent developer with a background in QA (an SDET) helping build out the infrastructure and other team members with various skill levels in automation contributing where they could – with oversight and mentoring from the more senior automators.
Thanks for the article!
Hi Sam. I guess it depends on your organization/team. But I agree with your who ever designs the framework regardless if they are a developer of qa they need to have real development skills.
Thanks for big pointers
Have to reflect on #2
If a developer creates your framework, it will have fewer design, scaling and maintenance issues and not crash. But if a tester wrote it, it will have support for testing paradigms, negative testing, assertions, and other domain solutions like oracles built in, and crash sometimes.
It needs a mix, developers make poor testers, and testers make poor developers for a reason – imagine the case of the backup feature in your app not storing the 4th address line in a customer form. This is not a problem for developers to deal with, it’s minor enough to not be a costly bug. But if the regression testing failed to catch that an annual backup/restore as part of product upgrade looses some customer form data, that can hurt the business 2 years later. This is a tester problem, not something we bother developers with because they must stay focused on the happy case and get product out of the door. Testers are best when in paranoid mode, the tools must mimic this outlook :-)
I have seen testers produce pretty poor test code, but that’s because I was a developer once myself and use my spidey sense to know when to swap hats on a daily basis. It also ignores the fact that developers actually do a lot of testing, and I kinda hope nobody fired the manual testers.