The testing dead have risen, and they’re hungry…consumed with an unnatural desire for bad automation practices to feast on. Your test automation efforts are crumbling, and they’re coming for you next. What do you do?
Matt Barbour, Senior Director of Engineering at Comcast Silicon Valley, shares with us some lessons learned about end-to-end testing from Zombieland. Listen up and discover how to escape the coming test automation zombie apocalypse.
About Matt
Matt is the Director of Test Engineering at the Comcast Silicon Valley Innovation Center, Matt Barbour has served in a number of engineering roles at the company since 2007. He and his teams are responsible for ensuring a top notch customer experience at Comcast’s large scale for set-top boxes, mobile devices, and browsers. Matt is a primary maintainer of a number of repositories on Github including Zucchini which enables highly concurrent Cucumber testing and Magic-Wand which takes all the complexity out of creating a Selenium test environment. Matt and his family live happily in zombie-free Half Moon Bay, CA..
Matt also recently spoke at StartEast on End-to-End Automated Testing: Lessons from Zombieland.
Quotes & Insights from this Test Talk
- PARM stand for P, performant. It needs to be accurate, it needs to give you something relevant as a result and that’s the A for accurate. Additionally, it needs to be repeatable, so you need to be able to get the same results each time and that’s the R. Of course, at the end even if you have a performant, accurate, and repeatable test if it’s not meaningful it’s a waste of a test and so that was the M, so P-A-R-M, PARM. In the course of my talk I was using that as a vehicle to say this rule helps you focus on this part of the PARM matrix for lack of a better term.
- What I kept trying to stress is writing automated tests is a development task and really there’s rigor around development and really you’re writing automated tests … This should be as good or better than the code you’re testing.
- One of the things I always like to stress to people especially when writing BDD is to favor DAMP phrases over D-R-Y which your natural inclination as a developer is to not repeat yourself. By doing interesting things with scenario outlines and Cucumber and things like that or just in general writing code that makes use of encapsulation and things like that and inheritance, but the reality is in order to increase the meaningfulness and accuracy of your tests it’s really a lot better to actually describe things in what we call DAMP phrases which is descriptive and meaningful phrases. With BDD it’s really obvious when you do that because your BDD scenario’s become a lot easier to read and understand and the whole thing … You’re able to pass the test of what is this test doing and why do you have it. It’s very clear.
- Actually I think Applitools does a really great job of solving the image comparison problem for situations where it’s merited. I still shy away from it where I can, but it actually has … If you can do it well it really yields great results and so I think when I’m talking about image comparison I’m talking about the thing that a lot of people do which is just bit comparison. You take a bitmap or a PNG and compare that to a known PNG just bit by bit and then you set some thresholds and say if it’s 90% successful then call this a pass. The Applitools tools solution is has definitely solved a lot of the problems around image comparison namely image maintenance, intelligent comparisons that can handle slight shifts between things. They really do a good job with that.
- I believe wholeheartedly in contributing and just being a member of the open source community. I think I really strived for our team to … Anything that we’ve done internally that has application outside of what we’re doing I really stress the team to get it out there and open source it because I really believe that we make use of a lot of open source stuff. I feel that a lot of the stuff that we do would be helpful for other people that are in a similar situation.
- I think the best probably advice I can give you, to anyone who’s struggling with this is you’re dealing with the real world and chaos in the real world is just exactly that. There is no way to escape it completely. Really the best way to deal with this stuff is … I heard there was a saying that Bruce Lee had a long time ago which is just be like water. It’s like you’re not going to stop a river, but you can divert a river. I think if you take that and say, okay, what’s my river in my test that’s a great way to think about it.
Resources
- Zucchini – sits on top of cucumber-jvm and maven to provide higher testing throughput by enabling concurrent test execution.
- DAWG – is a device library, which allows for device viewing and device reservation
- Magic Wand – is a helper for creating WebDrivers. This allows test to easily rely on different WebDrivers depending on different system requirements.
- Comcast Github
Connect with Matt
- LinkedIn: matt-barbour
- Company: Comcast Silicon Valley
May I Ask You For a Favor?
Thanks again for listening to the show. If it has helped you in any way, shape or form, please share it using the social media buttons you see on the page.
Additionally, reviews for the podcast on iTunes are extremely helpful and greatly appreciated! They do matter in the rankings of the show and I read each and every one of them.
Test Talks is sponsored by the fantastic folks at Sauce Labs. Try it for free today!
