83: Joe Colantonio: The Test Automation Playbook

20 December 2015, 01:47 PM

By Test Guild

The Test Automation Playbook

Today’s episode is going to be a little bit different. Since we’re nearing the end of the year, I thought I’d share with you the Test Automaton Playbook presentation I gave at the Øredev Conference in Sweden this past November.

I’ve had the privilege of interviewing some of the biggest names in the testing and automation field in 2015, and I think my presentation summed up all the lessons and good practices I’ve picked during those interviews.

One of my goals at the start of 2015 was to get my feet wet in public speaking by presenting at a conference. It’s something I’d been wanting to do for a quite a while, but being a naturally introverted guy, I was honestly scared to death to speak in public. I was about to let this goal slide until I received a speaking invite from a Einar Valgeirsson a member of the program committee for the Øredev, one of the largest developer conferences in Europe.

I took the invite as a sign that it was time to go for it – no matter how bad it may turn out to be. This is an actual recording of my first ever presentation. I actually had fun doing it, and I think the audience learned something from it. I hope you like it.

Quotes & Insights from this Test Talk

For Test Automation you always need to make sure that you're setting your managers' expectations. This is key because if you don't have your managers support for automation, a lot of times they don't realize that it's a development effort. They need know it's a development effort because development efforts, you need to maintain it over time. If you don't maintain you test over time, it's going to be a nightmare to reflect it, so managers need to know this. Managers sometimes come up with crazy metrics like 90% of our test of everything is going to be automated. We are going to have 90% automation and you have to basically do some push back and educate them that it's not necessarily possible and necessarily a good thing.
As your developers are committing more and more to the code base, you want to run tests really quickly and give them the feedback, let them know whether or not your test failed. If your tests are not testable, if your code is not testable, if your waiting to enter your sprint code or automate it, you are going to be in a big, big trouble there. You really need the developers to be thinking about testability from the beginning, from the requirement stage. Thinking about, “How could we make this more testable? How can we make it more automateable? Can I expose an API that a tool can use later on to help interogate this application? I'm I using unique ideas that will help my test tool interact with this application better?” Testability is a key to being successful on automation.
Readability. Once again this may seem trivial, but if you work with sprint teams, 8 to 10 sprint teams, if everyone is running tests their own way, they are using their own method names, they are using their own approach, you need to maintain your framework, it's going to be a pain. If you are contributing to the framework, if you see a method that exists that does something you don't understand what it is, you going to want your own method. Over time you have 8 to 10 sprint teams having their own methods that do the same thing, because no one took the time to read it. It wasn't really readable. That is going to be a problem because you want to keep things dry. Don't repeat yourself, have one place that you do changes, when you have multiple things doing the same thing, that becomes a maintenance nightmare.
Here's another great point. A lot of teams don't do this. As we do any automated tests, you want to make sure you're able to see them fail. If you don't see them fail, how do you know that when they're passing they're really passing for the right reason? This is like test rim in development. You can create your automation framework in a way that you used TDD for your development efforts. Make sure you're seeing that fail so when it does pass you have the confidence in knowing that's it's really going to catch real issues.
You really want to make sure, like I said that, before you interact with something, you interrogate it, you want to make sure that it's present. Before you manipulate it, make sure that it's clickable. Don't make any assumptions, always be proactive when you're writing your test to make sure that you're checking to make sure they're in a known state before you continuing. This will drive you crazy if you don't do it because sometimes it will pass, sometimes it will fail and you won't really know.
Meantime to diagnosis is really key. When the test fails you need to know, why did it fail as quickly as possible, so using some sort of logging mechanism. We started messing around with something called ElasticSearch, which is a really cool tool that allows you to see flakiness over time and gives you a nice dashboard view of seeing how your tests are behaving.

Test Automation TestTalk's Resources

Connect with Joe

May I Ask You For a Favor?

Thanks again for listening to the show. If it has helped you in any way, shape or form, please share it using the social media buttons you see on the page.

Additionally, reviews for the podcast on iTunes are extremely helpful and greatly appreciated! They do matter in the rankings of the show and I read each and every one of them.

Test Talks is sponsored by the fantastic folks at Sauce Labs. Try it for free today!