I’ve been involved with test automation for over 15 years, and I’m amazed that to this day there are misconceptions about test automation that most managers are not even aware of.Lets face it your boss doesn’t know test automation, but sometime they act like they do. This needs to stop.
Here are the top five things your boss probably doesn’t understand about test automation, and why you need to educate him or her ASAP:
1. Changing test tools is not going to solve your automation reliability issues.
It never fails. When it comes to crunch time for a release and our automation tests are acting unreliably, I always hear grumblings that the issue is with our test tool.
This was one of the main reasons a team I worked with moved from QTP to Selenium. For some reason, the boss thought that switching would improve their automation efforts and automation tests would start running reliably without much effort.
Recently this same team started blaming the test tools again for their test being unreliable and the manager wanted to look into changing tools. His suggestion? Wait for it – UFT! No joke.
It’s like saying the application you created in C# sucks, so rewriting it in Java will solve all your problems. Totally stupid. The problem is not your test tools – it’s your teams.
Changing tools is not going to solve any team dysfunctions you might be dealing with that are causing your automation efforts to be either ignored or neglected.
2. Test Automation is not just a QA activity.
This one will definitely get you into trouble if you don’t set your boss straight right away. When a team is involved in creating an automation framework, it becomes a software asset — just like any other development asset that needs to be maintained. Automation should also possess the same standard of code quality and coding standards. Accordingly, you should have your developers develop your automation framework — not a QA resource.
I don’t think many managers understand this. They believe that because it’s testing that it’s just a QA activity, so testers should be the ones who build the automation framework. And if you have a lot of QA people that have never written any code and you kind of throw them into that environment, you should expect that you're not going to get a quality framework. You are basically asking them to develop an application that tests another application. Since it is software development, why would you have a QA resource and not a developer to handle it?
3. Not everything should be a UI test
If your boss wants everything to be a UI test, watch out. If you're currently running hundreds or thousands of automated UI tests, it’s probably an indication that you're trying to do too much with Selenium or QTP. Based on my conversation with my TestTalks guests, I know of a number of companies that do have very large Selenium/UFT test suites that are actively trying to pair that number down and move a lot of that testing into unit and integration tests that don't require the full browser stack.
If you are in this situation, you need to educate your boss as to why a lot of those tests need to be gotten rid of. You should also take a long, hard look at why your manager believes that everything should be tested through the UI layer.
A helpful resource that offers info about a good ratio of end-to-end test vs. smaller integration and unit-type tests is How Google Tests Software.
4. Test automation does not replace manual testing
No matter how good your test automation scripts are, they will never replace someone that does some sort of manual/exploratory testing against your application. It’s always going to be a combination of very high-end intellectual manual testing and test automation. It’s a combination.
Simply having automation tools, or a large percentage of your tests automated does not equal quality. In my experience, many managers think that since they’ve made a large investment in either developing an automation framework or purchasing some test tools that it is a cure-all for their quality issues.
That is never going to be the case. Automation can only test what you tell it to. But the intellectual components of testing — really understanding your application’s design, and exploring the application with a tester’s mindset — those are the real intellectual elements. Tools will never be able to do that for you.
I’ve written another post that talks about this concept in more detail, called “Can automation do actual testing?” where I interview Richard Bradshaw, AKA Friendly Tester about his automation in testing concept.
5. Faster is not necessarily better.
Just because you have lots of automation doesn’t mean your application quality will be better. Just the other day I was talking with the manager of another team, and he showed me an executive dashboard that showed his project has 98% of all tests automated. He was really proud of that, but it scared the heck out of me. I’m guessing these tests either suck at actually testing important things, or aren’t really doing anything. To me this metric points out negligence, not quality.
That conversation actually reminded me of a one I had with Gojko Adzic, and some of the ideas he covers in his book Fifty Quick Ideas to Improve Your Tests. Gojko mentioned that what he often sees, especially working in teams that are stuck with a ton of automated test they can't maintain, is that they've done a “fast food” equivalent of testing; meaning they've written a bunch of things that were really easy to write and automate, but are ineffective in uncovering issues with their application.
So, sure — in theory, they may have 98% of their tests automated and can run their test suites faster than they could before, but those factors alone don’t make them better. As they accumulate, they start suffering more and more, because when you have tests that are hurting, automating them only makes them hurt more often.
Changing Your Bosses Test Automation Mindset
The point of all this is not to ridicule managers. In fact, the problem is not your boss – ultimately, it’s you that’s the problem (me included) because it shows that we as testers haven’t done a good enough job of educating management as to what the benefits of test automation are (and are not).
Yes – your boss doesn’t know test automation, but we need to stand up and correct them when they make ridiculous demands, and explain to them why it doesn’t make sense. Unfortunately, we often take the easy route and humor them by following their ill-advised wishes, or by gaming the system to prop up metrics with bogus numbers that give management a false sense of security.
Instead, we should focus on and care about what really matters: the quality of our applications, and ultimately, our application users.
