122: WalmartLabs Test Armada with David Cadwallader

• Basically, a few of my colleagues and I took it upon ourselves to do a little weekend hackathon projects and try to put together some automation for testing checkout across multiple browsers, and we made some good progress, and then we just gained traction from there. Eventually, that led to me having my own team building a whole organization around test automation, and then starting to roll that out into multiple parts of the Walmart ecosystem. Now, it’s actually a part of our continuous delivery pipeline. It’s a mandatory part of what we do every day.
• For the managers and the VP level persona, we really pitched this idea of continuous delivery. At the time, when we started this effort, Walmart had release cycles that would sometimes be 1 month apart, 2 months apart, and whenever we needed to do a release, there was just always a massive amount of stress, and coordination, and getting everybody on to this consolidated release train where we had to have people staying up till 4 in the morning doing a test cycle during off hours when it was less likely to affect customers, deploy something.
• One of the goals with TestArmada is we didn’t want to be just another test automation library, so we didn’t try to reinvent that wheel. We internally chose to use Nightwatch as our testing library of choice because we’re a Node JS shop so everybody is already thinking in Node at our company and because the kind of syntax of Nightwatch was … we thought was very familiar or very easy to grasp for developers and some QA people alike, and we had to think about the persona of a QA person who didn’t have strong development experience, and so they might not necessarily use, or appreciate, or understand immediately some of the more advance JavaScript constructs around promises, and things like that, and callbacks.
• When we first started rerunning failed tests multiple times before reporting their failure you should understand that when we first implemented this, we felt shamed. We felt ashamed of ourselves. This is something that as an engineer, you don’t want non-determinism. You don’t want errors that you feel like you can’t control, and we spent months really trying to pin down all of the different sources of flaky behavior in Selenium, and we talked to Sauce Labs about it. We talked with people in the community, and it was just a game of whack-a-mole, right?
• It’s super important for us to not have false positives is because we have shifted to relying on test automation as hard gates as we call them for everything from deployments to pull requests, right? We believe firmly in this shift-left or as we like to call it, “Close the barn door before the horse bolts.” We run a cross-browser functional test suite of the entire smoke test suite for an application using Sauce Labs, using TestArmada on every single pull request.
• I think my one piece of advice would be as developers, it’s very tempting for us to think about writing our tests from a developer’s point of view and thinking about, “Can I test with everything that’s going on under the hood? Can I test every technical aspect of my page?” Sometimes, what we’ve seen is that this really gets test writers and developers tripped up in this state of, “I’m trying to test something really complicated, and I can’t wrap my head around it,” and so what we really just try to steer everybody towards is test things that a user would test. Test things from the perspective of a user of the website.

Resources

• TestArmada
• Elements of Successful Massive Scale Automated Testing

Connect with David Cadwallader

• Twitter: @geek_dave
• Blog: WalmartLabs
• Company: WalmartLabs

May I Ask You For a Favor?

Thanks again for listening to the show. If it has helped you in any way, shape or form, please share it using the social media buttons you see on the page.

Additionally, reviews for the podcast on iTunes are extremely helpful and greatly appreciated! They do matter in the rankings of the show and I read each and every one of them.

Test Talks is sponsored by the fantastic folks at Sauce Labs. Try it for free today!