So, you might be saying, “Come on Joe. What can we learn about test automation from Zombie movies?” At first I wasn’t so sure myself, but after speaking with Matt Barbour about his StarEast presentation titled “Lessons Learned on End-to-End testing from Zombieland,” I’m convinced that the test automation zombie apocalypse is upon us.
So Why Zombie Movies and Test Automation?
Matt confessed that the title was a hook to grab people’s attention, but when he started thinking about end-to-end testing on devices, he realized we’re dealing with the real world chaos that ensues upon leaving our nice, safe component-level testing world — which led to him coming up the “zombie apocalypse” analogy.
End-to-End testing is unpredictable in terms of what things are happening and what things you can control. A lot of it cannot be controlled, so you have a strict code of rules to follow that will guide you through the chaos and, hopefully, bring you through to the end with a good set of working tests.
Create Zombie-Free Test Automation
A great framework to help your test automation efforts is what Matt called the PARM Theory, which is something that gives you some guidance around what you need to focus on in order to make your tests worthwhile.
• P – your tests need to be performant
• A – your tests need to be accurate
• R – your tests need to be repeatable
• M – your tests need to be meaningful
Rules to Follow for Test Automation
Don’t tolerate automation mediocrity. Signs that your tests are mediocre are:
• Low test pass rate
• Slow running tests
• You’re not sure what a test is doing
• Flaky Tests – they are passing and failing randomly
Mediocre tests can be caused by any number of things, including not using the right approach or methodology when creating your tests. Things like record and playback, using image comparison and not having a test data management strategy will leave you vulnerable to a zombie attack for sure.
Automation Testing is Development
I’ve covered this point many times in my blog posts and podcasts, but for some reason many teams still treat test automation as strictly a QA activity. But the fact is that you’re developing a piece of software to test another piece of software, and because of that automation is software development and writing automated tests is a development task.
Writing automated tests should be treated no differently than developing code for the applications under test. To avoid zombie creep in your automation efforts, you can fight mindless tests using good development practices like:
• Test automaton code reviews
• Using design patterns
• Making use of static code analysis
More Ways to Survive the Zombie Test Apocalypse
To learn about more ways to create reliable test automation, be sure to check out my full interview with Matt Barbour: How to Avoid a Test Automation Zombie Apocalypse.
Full Interview Transcript
| Joe: | Hey, Matt. Welcome to TestTalks.
|
| Matt: | Thank you for having me.
|
| Joe: | It’s awesome to finally have you on the show, but before we get into it could you just tell us a little bit more about yourself.
|
| Matt: | Yeah, so I’m the Senior Director of Engineering for Comcast, Silicon Valley. I’m responsible for applications on Comcast’s X1 Platform which is a cloud-based guide. The applications themselves are cloud-based and my background is … I actually came from the semiconductor test industry. I was formally working at a place called Teradyne and some other startups around that and I switched to software and now I’m here.
|
| Joe: | Awesome, yeah. Today what I’d really like to focus on is your presentation you gave at STAREAST this year about lessons learned on end-to-end testing from Zombieland. Why zombies is the first thing we should start with. What can we learn from automation from zombies or zombie movies?
|
| Matt: | Yeah, yeah. I got to admit the title’s a little bit … It was a hook, but where I was thinking was so end-to-end testing on devices, you’re dealing with the real world chaos that ensues when you get out of your nice, safe testing, component level testing world. I was thinking what’s a good analogy to that. I was like, well, it’s like a zombie apocalypse. It’s unpredictable in terms of what things are happening and what things you can control. A lot of it you can’t control, so you have to have a strict code of rules to follow that will guide you through the chaos and, hopefully, arrive at the end with a good set of working tests.
|
| Joe: | Yeah, I definitely agree. There’s definitely a lot of chaos in end-to-end testing. Pretty much every automation framework I’ve been involved with that’s end-to-end testing is chaos. You then go onto have a bunch of different rules and they’re funny, the different rules you have. The first thing that you go over though is the PARM theory, so what is the PARM theory?
|
| Matt: | The PARM theory is my attempt at trying to come up with something that gives you, I don’t know how to put it, but like a guidance around what do I need to focus on to make my tests worthwhile. It’s influenced by the CAP theorem in distributed computing which states that in any distributed system you can either be consistent, available, or partition tolerant, but you can’t have all three. It’s not possible to be completely one or all three of them. I was thinking about that, well, how does that apply to end-to-end tests and as it turns out end-to-end testing in real devices is … It has actually a lot of lessons to be learned from distributed computing and just dev ops in general.
|
| I said, well, what makes an automated test really useful. It definitely needs to be
| |
| able to be run fast, so that’s the P, performant. It needs to be accurate, it needs to give you something relevant as a result and that’s the A for accurate. Additionally, it needs to be repeatable, so you need to be able to get the same results each time and that’s the R. Of course, at the end even if you have a performant, accurate, and repeatable test if it’s not meaningful it’s a waste of a test and so that was the M, so P-A-R-M, PARM. In the course of my talk I was using that as a vehicle to say this rule helps you focus on this part of the PARM matrix for lack of a better term.
| |
| Joe: | Awesome, and I think it’s a really great model actually to help people like you said focus on … to see what they’re working on and make sure that they’re addressing all these areas. I guess you’ll probably get to it later on, but meaningful, what do you mean by meaningful? Is this so you don’t have a bunch of extra tests that really aren’t doing anything and you have a bloated test suite?
|
| Matt: | Yeah, yeah. I think the best way I could describe that is when you’re writing tests there’s stuff that’s easy to do. A great example is if you have a class that’s got getters and setters. You can write unit tests that check your getters and setters, but the reality is that’s probably not getting you very much in terms of actual meaningful test coverage. I’m sure you can make those tests all other phases of the PARM matrix. You can make them performant and accurate and meaningful and all that kind of stuff, but at the end of the day are you really gaining anything from this thing.
|
| That was what I was hoping to focus on with that was … If you extrapolate that out to an acceptance level test and our teams we uniformly … In our office we actually use Cucumber in an open source framework called Zucchini that allows us to go horizontal and we’ll talk about that later. The best way I can describe it is when you’re doing acceptance tests and you have feature files and things like that a lot of times people end up writing tests in such a way where they either try to do too much or not doing enough. That’s where the meaningful part comes in in my mind at least.
| |
| Joe: | The next thing you get into is know your way out of it or to aim for excellence and not tolerate mediocrity. Once again a lot of times I think it’s easy for teams to come complacent with test automation and it just becomes a checkpoint on a list of to do things to complete their sprint, but how do we avoid that?
|
| Matt: | Yeah, this is really the one that I would say is probably the hardest one to do because it’s a cultural change. It’s basically you’re looking at … The way I was describing it in my talk was there’s a phrase that you might hear periodically that perfectly describes this phenomenon which is that’s the way we’ve always done it. I’m positive everyone who’s worked for any amount of time has heard someone say that as an example of why they’re doing something. To me that is one of my least favorite phrases in the English language because it basically implies that what we’re doing is good enough and there’s no reason to improve it. I think there’s hardly ever a situation where that’s really actually the truth.
|
| Even if you haven’t said that or heard someone say it in your team there’s other symptoms of it like a low pass rate or your tests don’t finish quickly enough or like I was saying before if you can’t tell me what this test is doing and why you have it, things like that. To me the thing there is if you have these symptoms it leads to an engineering team that as you said just ends up checking the box saying we have automated tests. You can get to that situation where you have 100% unit test coverage and they don’t actually do anything. Or one that happens a lot, it actually happened a lot in our org before we started focusing on this was you get to about 75% pass rate, 85% percent pass rate and you’re like hey, that’s not bad.
| |
| The reality is you still have 15% of your tests that are failing and you don’t know why. It’s like now we got to respend an hour per test or hour per week or hour per day or something just trying to debug why test X failed. It’s just such a waste of time that had we taken the time upfront to do the tests in a way that was really well done … The theory here and I believe it’s been bouted, but what my experience is is that you spend the time upfront and you’ll reap the benefits later on because you’re not spending all this time debugging all these false failures. You can believe in the results that you’re seeing.
| |
| Joe: | Absolutely, and there’s so many things here that you just mentioned that I agree with. It is a cultural issue and every company’s different. Culture issues are the hardest things to fix, so I definitely agree with you there. There’s one thing that you brought up that … When someone says this is the way we’ve always done this and I know with larger companies I’ve seen this more often, but maybe it’s just because of my experience. I was just reading something recently about agile is not a noun and you brought up a point where your process should change over time, if it doesn’t then there’s something wrong. It sounds pretty similar to what you’re saying about your testing. That things change over time and you should be looking at it. It shouldn’t always be the same because if they are then there’s something probably wrong there.
|
| Matt: | Yeah, yeah, and if your tests are working the whole time then it means your product is probably not advancing either.
|
| Joe: | Right, so as we’re testing, we’re doing automation, how does the buddy system come into play both with zombies and with test automation?
|
| Matt: | Okay, so before my current role I was just director of test engineering. We actually had a QA org in our office and that was great. At some point we decided to flatten the org and say we don’t have a dedicated QA org, we embed our QA engineers with development teams and as a result I ended up taking ownership of quite a few things end-to-end. What it got me thinking about was a lot of times when I was just dealing with things on the test end, a thing we would always have trouble with is we’d have these really great software engineering test engineers. A lot of times they would have this inferiority complex saying … It’s like, well, I’d ask them to do something or I’d ask him to look into certain things and it would be like … You got the impression that it was like, well, that seems like a development task.
|
| What I kept trying to stress was yeah, writing automated tests is a development task and really there’s rigor around development and really you’re writing automated tests … This should be as good or better than the code you’re testing. Going back to my days in semiconductor testing it was if you’re testing a clock at 100 megahertz unit and looking for jitter in the picosecond range you need something that can measure a clock at 200 megahertz in [inaudible 00:09:27] second accuracy. It’s like if you don’t have that you’re destined to not get what you really want out of development or out of the tests.
| |
| The thing I was trying to espouse with this was yeah, writing automated tests should be treated no differently than developing code for the application under tests. To me that means if you’re a good test developer you need to know design patterns. You need to do unit tests, static code analysis, code review. A lot of these things that are out there are just accepted practice in the development world. A lot of times I don’t see those occurring in the test world and I feel it’s a major gap that you need to overcome in order to really be able to write really great tests.
| |
| Joe: | Once again I 100% agree with this. Test automation, you are basically developing a piece of software that tests another piece of software. All right, how about check the backseat and, I don’t know, I haven’t seen many zombie movies, but are there always zombies in the backseat that you’re not aware of.
|
| Matt: | They kind of pop up in the back and driving and grab you. Yeah, so this one you’re bringing only an unwanted passenger in the same way you can do the same thing with dependencies. The starting point of this whole thing is if you aren’t using a dependency management tool for whatever language you’re writing your tests in you probably have a problem here. Things like Maven or MPM or … I’m sure there’s a dependency management system for pretty much every language that’s out there. [Inaudible 00:10:53] is one. I feel without that you’re just asking for problems. Even if you do have that a lot of them allow for transitive dependencies where you say I depend on this artifact, but this thing depends on blah, blah, and blah. As a result those things can overwrite whatever you’ve specified in your dependency tree.
|
| This is another lesson learned from dev ops practices and good distributed
| |
| computing world there. In order to get repeatable deployments of software you need to lockdown everything, dependencies, everything, it must be 100% repeatable. To me it’s a similar problem with tests which is if you’re allowing some of your dependencies to override whatever you know you need in your tests you’re just asking for problems because what if they change theirs that overwrites yours. Now you’ve got this test that you haven’t changed anything, it’s been working fine and all of a sudden now it starts breaking. You’re just inviting unwanted passengers in your car so to speak.
| |
| Joe: | Awesome, and to just make sure I understand this correctly. I’m actually guilty of this. We use Maven and are you saying you should watch out before … Sometimes I’m just like oh, there’s a library for that. I’ll just pull it into my palm and have a dependency there, but you do this too many times it gets bloated and then maybe dependencies … Even though Maven’s supposed to be taking care of this in the background you may have different dependencies that are overwriting other files that they use that you’re not aware of.
|
| Matt: | Yeah.
|
| Joe: | Okay.
|
| Matt: | A great example, this happens a lot I’ve found is with a patchy HTP client, just the standard HTP libraries a lot of people use, a lot of times you’ll pull in some dependency that seems normal. Maybe you’re depending on version two of HTP client and for some reason you’re not forwards compatible with version four yet or for whatever reason. If you happen to pull in some dependency that is compliant with HTP client four or whatever that version is going to be higher than the one you have specified in your palm. Without locking that down it’s just going to overwrite it and say oh, he probably wanted the newer one because he didn’t say explicitly give me this version. It’s going to end up pulling in this version four of HTP and now your [inaudible 00:13:08] are just going to break.
|
| Joe: | I guess the next one, the next rule is cardio and I like this. It says looking for opportunities to obtain useful test metrics outside of traditional tests. Metrics are tricky for me, different teams sometimes they’ll engage in metric if you give it to them. What is a meaningful metric, too, then?
|
| Matt: | Yeah, so what I was thinking with this was a lot of times you’ll have a requirement that says something like … Memory use is a good example. Maybe your application shouldn’t ever go beyond 80% memory utilization in the course of normal usage. That could be defined … That seems like a requirement that would be reasonable for someone to specify for some product. You can go and actually write a test that goes and exercises the product and is monitoring memory the whole time.
|
| Another way you could actually achieve a pretty similar if not the same result is to actually just run your functional tests that you will presumably have written for this thing which generally exercises the product in a way that would be considered normal. Then as part of those tests just have something that’s monitoring the memory on this thing over time and as a result of that you can actually skip writing that additional test because you have the data already to validate that.
| |
| Joe: | All right, so the next one is limber up, make sure your test tools scale. I guess it says make sure they go parallel in both directions, so what do you mean by that?
|
| Matt: | A typical problem a lot of people have … I would say probably anyone who’s doing web or mobile development is just your like device compatibility test. You got to make sure this thing runs … If you’re dealing with a mobile app, for example, it has to run on an android and an M and L and IOS, all these different versions and different makes and models. You have this big compatibility matrix of these are all the make and models of things I need to test on. Especially with mobile it’s just an astounding list, so right off the bat you’re probably going to have to pair it down. Even if you don’t running tests on these things, if you’re not doing that in parallel you’re just signing up for just an endless run of just serializing these things or doing some weird … using parallelizing, but doing it in a way where you’re using your Jenkins or TeamCity or something and hacking it that way.
|
| Really to me the elegant way is to actually have your test framework take care of that for you. That’s actually why we wrote Zucchini which is an addition to Cucumbers. Actually, on GitHub. GitHub.com/Comcast/Zucchini, but basically it’s write your Cucumber tests once for one device. Once you have that working the framework actually just allows you to add any number of additional devices in parallel and without any additional work it just kind of works. One of the things that … Beyond just device compatibility tests though being able to do things in parallel it’s really a powerful tool if you use it correctly.
| |
| Another example I like to use is let’s just assume that you have just a single device that you need to text against. One way you can actually get … I wouldn’t say it’s for free, but it’s cheap to get another useful metric here is run two instances of the same device. Run the same test in parallel on them at the same time and see what the results are. You should, if you’ve written your test right, you should be able to see that those test results are identical because theoretically you’re not susceptible to … If you ran them in serial you might … A service that you depend on might be behaving slightly different at the same time, you know what I mean. It’s like you get this free metric of how reliable is my test. If I can run this thing twice at the same time I might be able to know that this test is a little bit more reliable than something where you get one
| |
| passing, one failing or both failing or whatever.
| |
| I think once you’ve looked at that the next thing to look at is … I would call that parallel testing horizontal scaling where it’s just same tests replicated on N number of devices. Beyond that the next thing is to think about how do I run more tests in parallel. One of the downsides of doing horizontal scaling is that you’re still limited by how slow your slowest test is. Basically, you’re going to get a whole bunch of tests that execute slowly and you’ll get one long one and then a bunch of slow ones and then a long one and ultimately the sum of all those is your test time. If you were able to do something in a way where you pipeline them in such a way where you run test one and two on two different devices and you ran test three on a separate device, these are all the same device, you end up with … Your total test time it’s really determined by whatever your slowest test is.
| |
| Joe: | Yeah, absolutely. I definitely agree with that. I don’t know where I heard it, but I heard a … Facebook was just able to do it in such a way that their slowest … Whatever their slowest test was that’s how long it took to run the whole entire test suite and I don’t know if they used [inaudible 00:18:25] to just throw up instances of an environment for each single test, but that sounds like what you’re getting at. You need to be able to scale up, so you don’t have this bottleneck eventually.
|
| Matt: | Yeah, I think I actually saw somebody from Facebook present it at GTAC last year with that and it was exactly that. I think they had … As I recall they had an open source date, so I was a little bummed about that. It definitely seemed like exactly what you’re talking about where they had effectively an unlimited supply of devices whether they were simulators or real or whatever. That’s the one downside of trying to do all this is in order to be able to do highly parallel testing you really actually have to … You have to have effectively an unlimited supply of devices to test against otherwise you have to start making compromises.
|
| One other thing I wanted to touch on, too, was looking at the test tools that you use in general because they’re just making your test scale, but then there’s also making it so that your test authoring scales. There was a great example we had internally which was … I mentioned we’re in charge of applications for X1 and for the longest time the way you could automate tests against those was using a proprietary language and proprietary everything. It meant a whole lot of tribal knowledge and a lot of people that … It was basically like when you try to bring somebody up it would take months because they’d have to go ask the right people and read the right docs some which were out of date.
| |
| It was just a mess and so what we … [Inaudible 00:20:00] and I went to the Selenium Conference last year and we had an epiphany there which was what’s the industry standard for writing automated tests for mobile and web? It’s Selenium and WebDriver. That in and of itself is actually not specific to web and mobile. It’s actually just an API and so we said what if we could make it in such a way where we’ve got this proprietary way we can interact with the devices. What if we abstracted it in such a way where our test authors could actually write Selenium tests for this thing. It turns out we were able to actually implement that and the results have been great.
| |
| I think it’s really easy to find people that can write Selenium code and it takes a small amount of time to get them up to speed on just the technology in which we … the applications out there. They don’t really have to know anything more than that. We found really great success with that and I feel that’s one thing that a lot of companies that have that similar problem which I know that exists out there. I feel that’s a really great opportunity to really improve how well your tests scale.
| |
| Joe: | Is this how I think Appium came along is they just … Because Selenium uses an API instead of a JSON wire protocol. That as long as you can write to that protocol it doesn’t matter. Is that what you mean? You’re able to write to that wire protocol?
|
| Matt: | Yep.
|
| Joe: | Okay.
|
| Matt: | Yeah, that’s exactly right. The way Selenium works on just your browsers now and Appium as well is it’s kind of a server-client model where the browser that you have running on your computer or something has some proprietary way in which you would be able to write tests for it and they have these … Chrome, for example, they have a ChromeDriver that speaks Chrome’s native language and exposes the API and so then any Selenium client whether it’s Java or JavaScript or whatever else can actually just talk with that in a way that is agnostic to whether you’re running on Chrome or running Firefox or whatever else. It’s a similar problem with us in our platform as well.
|
| Joe: | All right, so as we’re scaling, as we’re running in parallel the danger here is and it comes to your next point is when something fails you need to know why it fails, so don’t be a hero. Is that something you mean by don’t be a hero? You need to know when it fails and why it fails and especially if you’re running a parallel. You can’t go it could be running because I have … It could be failing because I have 20 assertions, which one failed I don’t know.
|
| Matt: | Yeah, yeah. The [inaudible 00:22:33]. Actually, the gentleman who presented before me and I wish I could remember his name, I can’t remember it, but he had a great presentation around this which was basically if you’re trying to do too much in a test you’re probably asking for problems. A situation this happens a lot in I’ve noticed is when people are trying to just convert existing manual tests to automated tests, so naturally if you have a set of manual tests they’re going to be broken up into a few different steps.
|
| One of them will be [inaudible 00:23:03] the app and the next one is push button X and click button Y and then look for blah. Your natural inclination as a test automation engineer is to say, okay, those are concrete steps that I can implement and I have the ability to verify these things, so why not? I can verify the app once and I can verify that I pushed this button and that button. I can verify this on screen and that can work, but we’ll get to it later. I think depending on the test methodology you choose to do those things you could actually start running some real problems.
| |
| One of them that classically happens is image comparison. Your natural inclination is I can just do an image comparison, why not? Let’s assume that your image comparison success rate is 99%. If you have a comparison on each of those steps you’ve now compounded your … You’re basically saying each time you add one of these things, these image comparisons to your overall test you’re reducing the likelihood that the test is actually going to succeed just because of the nature of the beast with respect to image comparison. If you do four image comparisons in a row and each of them is 99% accurate you really … At the end of the day your test is only 96% accurate, 99% of 99 of 99 … just multiplying them out you end up with something like that. It gets a lot worse really quickly if your image comparison success rate is not at 99%. If it’s 95 or even 90 you’re down to 60% success rate at the end of the day, so it gets really bad really quick.
| |
| One of the things I always like to stress to people especially when writing BDD is to favor DAMP phrases over D-R-Y which your natural inclination as a developer is to not repeat yourself. By doing interesting things with scenario outlines and Cucumber and things like that or just in general writing code that makes use of encapsulation and things like that and inheritance, but the reality is in order to increase the meaningfulness and accuracy of your tests it’s really a lot better to actually describe things in what we call DAMP phrases which is descriptive and meaningful phrases. With BDD it’s really obvious when you do that because your BDD scenario’s become a lot easier to read and understand and the whole thing … You’re able to pass the test of what is this test doing and why do you have it. It’s very clear.
| |
| Joe: | That’s a great point. I definitely agree, DAMP or over DRY. How about travel light, so once we get over that we have our well-written future files and code. What does travel light mean?
|
| Matt: | This ties in to what I was just mentioning where even if you’ve got everything written well your choice of test methodology can really cause problems. I mentioned image comparison. Record and playback tests are just horrendous for that kind of thing. I think if you’re trying to do end-to-end testing using a record and playback tool you’re probably just wasting your time. Another one that we’ll get into next is just not controlling your data. I think there’s so many different ways that you can write tests. Even if you’ve followed all the other rules thus far, you can write a test in which it’s just a maintenance nightmare or it’s just unreliable to start from. There’s just so many ways you can shoot yourself in the foot with this stuff.
|
| Joe: | Once again the data management is difficult. I’ve seen with a lot of different teams that’s a really difficult piece. It’s scary how everyone I speak to has the same exact issues. Yeah, I definitely agree with this. I just need to mention this just in case. I know [Moshi 00:26:48] is someone that introduced me to you and I don’t know, explicitly [inaudible 00:26:52] and [inaudible 00:26:53] Applitools of visual validation testing, but is that what you mean by image comparison?
|
| Matt: | Oh, yeah, I’d be remiss if I didn’t mention them because actually I think Applitools does a really great job of solving the image comparison problem for situations where it’s merited. I still shy away from it where I can, but it actually has … If you can do it well it really yields great results and so I think when I’m talking about image comparison I’m talking about the thing that a lot of people do which is just bit comparison. You take a bitmap or a PNG and compare that to a known PNG just bit by bit and then you set some thresholds and say if it’s 90% successful then call this a pass. The Apple tools solution is actually really … It’s definitely solved a lot of the problems around image comparison namely image maintenance, intelligent comparisons that can handle slight shifts between things. They really do a good job with that, so I guess that one could be taken off the list if done correctly.
|
| Joe: | Awesome, yeah. My company, I’m still waiting for them to get a license. I’ve been using Applitools for a little bit now and I’m really enjoying it. I guess along the same lines as test data management you have another slide talking about control all the data in your system.
|
| Matt: | Yeah, so this is probably one of the tougher ones from an engineering perspective to do in a lot of cases because there’s a reason a lot of people don’t have mocked out data for these systems or it’s insufficient is it’s really hard to do. Depending on where your data’s coming from the harness that you use to generate the controlled data could be just as complicated as the system you’re testing. My strong opinion is that if you aren’t controlling this data you’re just asking to have unstable tests or even worse letting defects get through your production. I read a lot of books on dev ops and sight reliability engineering and what not. I find the topic fascinating.
|
| One of my favorite quotes from that … I think it came from a Google book on SREs and I think it applies here which is hope is not a strategy. If you just hope that certain things aren’t going to happen in the real world or whatever it’s going to happen and it’s just inevitable. A great example is we have an application that’s a sports application. Sports are a really great example of unpredictable data, live events. There’s rain delays, there’s overtime, there’s cancellations, just weird stuff happens. On top of that live events especially with sports almost always happen in the evening times, on weekends. Generally they’re not happening during working hours although there’s certain cases there are. If you’re just relying on real data to … If you don’t have a way to mock out that data you’re basically hoping that the Sharks Blues game goes to triple overtime just to see what happens with your app. It’s just asking for problems.
| |
| One thing I like to stress is that if you … You don’t need to do everything. You need to look at what your tests call for and where your tests either can fail or have gaps in terms of coverage and have something that can get you that data, but you don’t necessarily need everything. It’s going to be dependent on the application. It’s a case of start somewhere and start small and just control one thing and see if you can get that to work and then just start building up from it. I think at the end of the day you’ll be thankful that you’ve done this even though it seemed like just an inordinate amount of work. At the end of the day it’s so worth it to not have to worry about staying up on weekends and watching games that we might not even want to watch.
| |
| Joe: | Now one of the funniest points you make, Matt, is the beware of bathrooms slide. Can you tell us what is beware of bathrooms.
|
| Matt: | This is my favorite slide actually. What I mean by this is you’ve done all the heavy lifting on your test side. You really follow all the rules, you’ve got great tests that in your mind have been written well, they’re robust, but there’s things that can get in the way of performance of these things and namely the architecture of your system. Using the sports app example again, the frontend is separate from the data provider for the whole thing and it’s your typical MVC deployment. One of the things is that the data provider is actually a shared resource between multiple clients. If you’re writing a test … Normally you’re going to write a test and you’re going to start with one device and then once you have that working you’re just going to throw a bunch of devices on it and scale it out horizontally.
|
| In this situation you could be writing tests and you’re controlling the data and you have your games happening when they should happen and the scores are predictable and things like that and it’s going to work great with one device. You have one client connecting up to the backend and you’re changing the data back in all the time and whatever, it’s great. As soon as you throw a second device on there now you have two … You basically have two devices competing for the same resource. They’re both trying to change the data back in at the same time.
| |
| In this case the first thing I always try and stress to engineers is don’t necessarily
| |
| take for granted that the architecture is the right architecture. Start from the fact that, okay, I’ve identified this, this is potentially an architectural bug. The first thing is always to ask is the application architecture the right architecture for the product as a whole. Not necessarily saying it’s the best for tests, it’s really is the way it’s set up correct. That often times requires talking to some architects and lead developers and things like that and just making sure everyone’s on the same page.
| |
| Once you’ve arrived at that and by the way if you haven’t … If it turns you found an architectural bug these are incredibly valuable bugs to have found. Let’s assume that it’s not an architectural bug. Now you’ve got this situation where you’ve got to deal with this. There’s the elegant way and the brute force way. I think 5 to 10 years ago the elegant way probably would have been the right way to go which is to synchronize your tests in such a way where all of your things in parallel aren’t modifying the same data resource at the same time. That adds a decent amount of complexity to your tests because now you’re having to deal with gating between threads and things like that.
| |
| We actually added some capability to do that in Zucchini and then as a result … We started looking into using it and what we realized was huh, we’re deployed into … We use OpenStack for deployment and Amazon would be a similar thing, but VMs are effectively free at this point. They’re so cheap and it’s so easy to spin them up and tear them down especially if you’re working in a virtualized environment. That why not just stand up in duplicate incidence of your system such that it’s identical because we’re deploying it in a way where server deployments are repeatable and we can deploy N number of servers at any point in time and it’s really no big deal.
| |
| What if we just said we want to do 20 devices in parallel, why don’t we actually just stand up 20 different stacks? It’s short-lived, they’ll live only for the time that you’re running the tests, but the reality is now you don’t have to make your tests complicated at all. You can actually have the tests run … In the same way that you would have run on a single device you can run it in parallel and it’s really no big deal. I think cloud deployments are really great for that kind of thing. This is a case where if the elegant solution might earn you street cred, but really probably the easiest thing is to do is just throw hardware at it.
| |
| Joe: | Once again I 100% agree with you. You mentioned Zucchini and I think there’s a few other tools that you actually use at Comcast often. Could you just tell us a little bit more about tools that you recommend others taking a look at. I think you have open sourced some things that you’ve done at Comcast.
|
| Matt: | Yeah, yeah, so we try … I really strive … I believe wholeheartedly in contributing and just being a member of the open source community. I think I really strived for our team to … Anything that we’ve done internally that has application outside of what we’re doing I really stress the team to get it out there and open source it because I really believe that we make use of a lot of open source stuff. I feel that a lot of the stuff that we do would be helpful for other people that are in a similar situation.
|
| Zucchini is one, I mentioned that. That is Cucumber testing in parallel. By the way, all this stuff is just on GitHub.com/Comcast. Zucchini is one that we do that is core to almost all of our tests. Basically every test of my [inaudible 00:36:38] and actually all the tests in our office are using Zucchini as their means for executing Cucumber feature files and also scaling out horizontally. We have a device management tool that we created called DAWG, D-A-W-G. That one is really interesting because in our situation we’re dealing with set-top boxes most of the time although we have mobile devices and web browser stuff.
| |
| One of the challenges we had was just understanding what we had available on racks and certain characteristics of them and having a lightweight reservation system that could work with Zucchini and some of these other tools that we have that allowed us to have a shared pool of resources that anyone could jump on, play around with the device, and then return it back. DAWG actually has three parts. It has a meta data system, it’s got a reservation system, and then we have this thing called DAWG show, which is actually if you have the right hardware on the racks and I think we have it all listed on the [inaudible 00:37:52] site, you can actually interact with these things directly through your browser. It’s like a poor man’s sling box, but it actually works really well.
| |
| Then the other one … We’ve got a bunch of other ones, too, but Magic Wand is one that our teams that are doing Selenium type of testing specifically around using Sauce Labs in general, browser and mobile stuff. They use Magic Wand as a way to simplify getting and configuring a driver. It’s like you say in your test I need Chrome running on OS X 10.11 or something, whatever it is. You say that’s what I need and I don’t care where it comes from whether it’s a local device or whether it’s from Sauce Labs or whatever else and that handles that abstraction.
| |
| I think beyond all the stuff with open source I think there’s so much great knowledge out there. Especially if you’re using industry standard stuff there’s really a great amount of knowledge out there and beyond just test technology. I think I mentioned dev ops and things like that. I’ve really been inspired a lot by some of the work that’s been done in the dev ops area in terms of just learning what’s happening in there and applying that to testing. The commonality, it’s just so common of a problem which is deploying servers and writing automated tests on production devices is just … They’re both chaos and it’s really just a matter of figuring out ways to … You’re never going to avoid the chaos, it’s just a matter of figuring out ways to work with the chaos.
| |
| Joe: | Okay, Matt, before we go is there one piece of actual advice you can give
|
| someone to avoid an automation project zombie apocalypse and let us know the best way to find or contact you.
| |
| Matt: | That’s a great question.
|
| Joe: | Thanks.
|
| Matt: | Gosh, I don’t know that there’s one answer. I think the best probably advice I can give you, to anyone who’s struggling with this is you’re dealing with the real world and chaos in the real world is just exactly that. There is no way to escape it completely. Really the best way to deal with this stuff is … I heard there was a saying that Bruce Lee had a long time ago which is just be like water. It’s like you’re not going to stop a river, but you can divert a river. I think if you take that and say, okay, what’s my river in my test that’s a great way to think about it. Yeah, you can get a hold of me … Probably the best way is just via email, Matt_Barbour, B-a-r-b-o-u-r@Cable.Comcast.com. You can also get a hold of us if you have issues or things like that, if you ever want to use any of these things. We’re very eager to help if anyone wants to get going and they don’t really know where to start. You can always pin me and I’m happy to do my best to help out. I think it’s a community problem.
|
Hi Joe,
I need a code to capture tooltip.
I have used the following code but its not capturing the tooltip.When I hover over password edit field it should display tooltip.
Function Tooltip_check()
Dim myp,myo,tooltip
Set myp = Browser(“title:=.*”).Page(“title:=.*”)
Set myo = Description.Create()
myo(“micclass”).value = “WebEdit”
myo(“type”).value = “password”
myo(“name”).value = “pwd”
myo(“html tag”).value = “INPUT”
myp.WebEdit(myo).FireEvent “onmouseover”
‘wait 3
‘commented it as it didn’t worked
‘tooltip = myp.WebElement(myo).GetROProperty(“innerhtml”)
‘commented it as it didn’t worked
‘tooltip = Window(“nativeclass:=tooltips_class32”).GetROProperty(“text”)
‘used xpth as well but didn’t worked
tooltip = myp.WebElement(“xpath:=/html/body/script[10]/text()”).GetROProperty(“text”)
If tooltip = “This is a required field” Then
print “test case pass”
else
print “fail”
End If
End Function