Are you looking for an awesome mobile grid solution like Sauce Labs, but don’t yet have the budget for this type of service? Although using an existing Cloud-based service is optimal, there are times when (and reasons why) having your own in-house mobile grid setup can be useful.
Today we’ll take a look at how you can set up your own in-house mobile grid for your mobile test automation efforts with Justin Ison, a senior software engineer at Microsoft.
What is a Mobile Testing Grid?
Most of us are familiar with a Selenium grid, but what is a mobile grid? It's basically the same thing, except you're swapping out browsers for mobile devices.
Why Use an In-House Grid Solution?
There are many reasons why you might want to have your own in-house mobile grid.
For example, Justin started his own mobile grid solution because he was working for a small startup that didn't have a lot of funds allocated to run all their tests, with every commit, all the time because it would have been too cost-prohibitive for them.
However, he still wanted to run regression and quick smoke tests just to get exposure on more devices. That’s the main reason he decided to create his own in-house solution for test case execution.
Before we go any further, I just want to point out that Justin does not believe that this is an either/or decision. There are often times when you’ll want to run tests against your in-house grid, and others when you’ll want to use a enterprise testing tools service like Sauce Labs.
Justin and I agree that Test Cloud services are the future, and we believe you should leverage both.
Tools Used to Create a Mobile Testing Grid
If you’re looking to get started with your own mobile testing grid, these are some of the tools Justin recommends:
• Jenkins – for orchestrating your mobile tests
• Appium – for automating a device’s UI
• Allure – an open-source framework designed to create software quality reports for your test execution results in a clear format that everyone on your team can understand.
• Ruby – for programming
• Selenium Grid
Don’t Start from Scratch!
Rather than starting from scratch, wouldn’t it be cool if you could just leverage Justin’s mobile grid solution?
Wouldn’t it be great if there was a way that you could use what Justin has already done without going through all the frustrations and roadblocks he faced while creating his solution?
No worries! Check out Justin’s mobile grid implementation. (He was cool enough to post an example on Github):
More Mobile Grid Automation Awesomeness
I know it can be, super frustrating; I’ve been there. As with all other automation, if you keep at it you can eventually get it to do what you want it to. Just don’t give up and listen to the full episode for even more mobile grid automation awesomeness.
| Joe: | Hi Justin, welcome to Test Talks.
|
| Justin: | Hi Joe.
|
| Joe: | Great to have you on the show. Today I'd like to talk about a presentation you gave recently about the mobile grid, getting started with Android and iOS and also talking about your experience with test automation. Before we get into it, could you just tell us a little bit more about yourself?
|
| Justin: | Yes sure. I'm a Senior Software Engineer for Microsoft. Been in IT industry for about 17 years now both as a QA tester and software engineer. Most recently I now work for Microsoft, before Microsoft I worked for a small startup in Berlin, Germany named 6Wunderkinder makers of Wunderlist and then Microsoft acquired us last June of 2015, so that's how I became a Microsoft employee.
|
| Joe: | Awesome. How's that transition going, I guess I'm getting off topic already, but going from one company acquired by another, are there different tools and techniques you need to learn in automation or are you slowly assimilating into their style?
|
| Justin: | Actually no. Microsoft has been great. They have actually let us be ourselves, be still small startup. We've adapted to some of their tools, but for the most part we still use all the same open source tooling and languages outside of Microsoft's ecosystem.
|
| Joe: | Great. I guess the first point I want to make with your presentation is I guess a lot of people are familiar with the Selenium grid, but what is a high level mobile grid?
|
| Justin: | It's basically the same except you're swapping out browsers for mobile devices.
|
| Joe: | Cool. I guess a lot of people are also aware of cloud based solutions like SaaS Labs. When would you think its appropriate to have your own in house mobile grid as oppose to going with a cloud base service.
|
| Justin: | I'll give you a little history as to why I went to this approach. I had no intention to ever go this route. There's well established test automation cloud services out on the market currently and there are several in Europe as well. However at the time, Wunderlist was a small startup and we didn't have a lot of funding allocated in our budget to run all our tests all the time, with every commit to these tests services. However, I still wanted to run tests there whether it's a regression run or quick smoke tests just to get exposure to more devices.
|
| That's where the problems started. The tests were being sent to these services, they would sometimes hang. A test that should've taken only maybe two minutes to run would complete in fifteen minutes. Sometimes the just would just randomly fail or they would pass fine locally, so there was a problem, you only got half the picture.
|
|
| I couldn't tell if the device on their end was having issues, or the device was connected to internet connection. There's a whole bunch of reasons why it could happen like [inaudible 00:03:18] issues, so it was really difficult to know. That's sort-of what lead me to creating my own device lab. At the same time, I also didn't want to sacrifice what I got from these services like video recording, screen shots, logs.
|
|
| The other benefit that I found was you have full access to these devices so you can pull the device data base files for your app from that device if it's an emulator. There's a bunch of nice extra benefits you can get if you have it all locally, then you can tell right away what the problem is where as you don't know the full picture when you're running your tests on cloud services. That's why I buckled down and created my own sort-of internal test cloud.
|
|
| Joe: | Awesome. Where does one start creating their own internal test cloud? Can you walk us on a high level maybe the pieces of the thought process you went through to actually implement this?
|
| Justin: | Yes sure. I knew all along that these cloud tests services used a grid, a Selenium grid for their main architecture connectable devices, so I obviously started there. I started Googling around, which wasn't a lot of information out there to do this. You might find some information, but it might not be in your findings. On my Ruby developer, I couldn't find anything in Ruby. I'd find some stuff in JAVA, maybe some stuff in Node, nothing like I could figure out to put into my findings.
|
| Started from that, just Googling and then learning more about how the grid works, then just started from there. Connecting the devices, which essentially each device is a Node, connecting the Nodes to the hub, then you connect to the hub and pushing a test to my local hub server.
|
|
| Joe: | Great. Were there any tips or tricks you found help to make that transition smooth with that setup?
|
| Justin: | No, trial and error. It was a big process. Just learning it, sort-of you know, what we probably do for test automation is trial and error and just keep at it. It took me a while to smooth things through, but then I got it to run consistently at a nice performance and very little failures. Once I sorted out all the little nuances and bugs here and there.
|
| Joe: | One road block I can think of, this is just anyone moving towards a Selenium grid I guess would be, do you have any tips for people that have been used to running their tests in single thread and then they need to really start working and making the tests distributed running in parallel. A lot of times I find issues where the data tested as being held up by one test because no one thought about, hey, there might be another test running and then maybe interacting with he same data.
|
| Justin: | Exactly. Paralyzation brings on a whole other level of complexity. Especially with data sets like you said. Fortunately at Wunderlist we have an API, public API that we use. Every user on every thread is it's own user, creates his own data and then after that test is completed, it's torn down and deleted, the information is deleted. None of the data on the other tests is impacted by that user.
|
| Joe: | I guess another question that comes to mind is a lot of people use emulators, and one of the benefits that people get from using a cloud service is actual physical devices so they don't have to say, “oh I need a have an iOS or I need an Android with this flavor of marshmallow whatever.” How do you handle that, do you just use emulators or do you actually have a little lab set up?
|
| Justin: | Both. I run on both emulators and real devices. I'm okay with running all our smoke tests on emulators, because it's quick. It tells us right away if the build is stable and it's working, but I still feel like regression runs or further regression runs should run on real devices like either periodically like nightly or weekly depending on whatever your development cycle is, or your release cycle.
|
| Joe: | Cool. How do you manage hardware? Say there's a new update that's updated, are there any tips for people that are trying to do this on their own? Is it just a matter of you need someone that's actually staying up on it and keeping track of all the different releases and physically going in and updating all these devices?
|
| Justin: | Yes. That's basically what it comes down to. You'll get from the devices, you'll get an alert when the device needs to be updated, but you still want to control it and turn off auto-updates if you want to keep certain devices on an older version of the OS. What we do for emulators, I create an emulator for every single STK level that we support for our app and run all our tests against those in parallel. You still should do a subset of the same for your real devices.
|
| Joe: | Cool. I guess I also saw on your presentation something about video recording. That's one of the cool features of using a cloud implementation I think of the grid is, there's actually a video play back.
|
| Justin: | Exactly.
|
| Joe: | What tool am I using to do that?
|
| Justin: | I didn't want to lose any type of functionality features of the cloud service. I wanted to have everything they give you but have it still locally. For real devices on Android, you can actually, I think it's above OS 4.1, you can video record from the device itself and so you just do a screen cast, an ADB screen cast to the device and then you can pull that MP4 off that device and then attach it to your report. I'm a big fan of Allure and that's what we use for our reports.
|
| However for emulators, you can't video record. They don't have the codec to do it. Then I've created a little open source tool that does screen captures like every half second and then pieces them together and creates a video of it. Then I added a bunch of other helpful tools with it as well. I have a tool if anyone out there wants to use it, they can.
|
|
| Joe: | Will this only work with mobile or would it work with any selenium grid?
|
| Justin: | It's only for mobile. It's for Android mobile devices and emulators and iOS for real devices and simulators.
|
| Joe: | Cool. Another open source project I was looking at is selenium grid extras, which its not for mobiles, so that does have the video recording which is kind-of cool.
|
| Justin: | Yes, I've heard of it. I think the guys at Groupon did it.
|
| Joe: | Yes. How do you handle, or are there any differences when handling an application ran for mobile apps in your continuous integration environment as oppose to just a web app? Any thoughts around that?
|
| Justin: | It's almost exactly the same. We used Jenkins internally for the local lab and yes, it's just setting up a job for Android and iOS to compile and build. Once a change has been detected for, we use anything like as a master branch, we compile it and from that job we have a downstream job that will then do like a test set up for our mobile devices like, even for emulators, if you're emulators, it will actually launch emulators, get them ready. Once the emulators have started, it then has another downstream job that runs the test. Then once the test runs we have a downstream job to then close all the emulators.
|
| Same for the devices we have a downstream job to toggle WiFi on and off to reset the network connections. Basically if the devices is ready and then the same tests run on real devices.
|
|
| Joe: | Cool. You also mentioned earlier that you're using, I think you said Allure for reporting. I've never heard of that. Can you just tell me a little bit more about the Allure reporting feature?
|
| Justin: | Yes. Allure is an open source framework, test framework. It's almost in every single language out there. I use Rspec Ruby, but there's one for jUnit, end units I'm sure for C Sharp, I can't remember how many but it's getting more and more popular. I believe its a Russian company that built it. It's really nice because you can, it's very dynamic, you can customize it to the way you want it. You can attach different types of attachments in it, which is nice. That's what I do for ours. All the meta-data that I capture like logs and videos and screen shots, I can throw those all into the test report so I have them there for de-bugging after.
|
| Joe: | Very cool. This sounds like a cool idea and especially for the company I work for. For some reason they need everything in house and it's like a Federal case to get anything in the cloud. How realistic do you think this is for someone to actually implement on their own? What skills do you think they need in order to be successful with implementing their own mobile grid?
|
| Justin: | It's not, I mean, what I always tell people is once you know it, it's not that difficult. It's just finding the information to know how to do it.
|
| Joe: | Right.
|
| Justin: | I think it's just a little bit of dev-ops knowledge. Obviously Linux or Unix or base scripting would be helpful. It's not too bad. I actually have an example on my get hub as well for how to get started.
|
| Joe: | Excellent. Yes, I'll definitely have a link to your get hub and all your other links within the show notes, this would be a great resource. Am I correct, do you test the Wunderlist application, the to do list?
|
| Justin: | I do, yes.
|
| Joe: | Awesome, that's one of my favorite apps.
|
| Justin: | Oh good.
|
| Joe: | Any approaches you use for testing a mobile app like this as it flows or is a…do you break them out into modules? What's your approach for testing a mobile app like Wunderlist?
|
| Justin: | Yes, it's sort of like modules. It's almost basically very similar to like a web automation test approach. Just very small tests. Each test doesn't rely on another test to pass. I think the smaller the test the better. I think you should for good tests framework setup, you should always try to maximize all your tear ups and tear downs into common steps in those areas so that whey when you're tests actually run they're only running steps specific to what you need to do for the tests. Sort-of along those lines.
|
| Like I was saying, we have an API so I make use of our API a lot to do some actions that would reduce the amount of UI actions I would have to do because actions in the UI go into a test may actually end up breaking but I don't really care about going…that piece of the step of the process to get to the final result of the test.
|
|
| Joe: | Now that's a great approach. The API, is this something you developers already created or is this something you've asked them, hey can you build me an API so I can test this better?
|
| Justin: | Well actually no we have a public API.
|
| Joe: | Oh, gotcha, gotcha, okay.
|
| Justin: | Yes. Thankfully that's the approach that I use. I actually gave a talk similar to why I went this route and I actually, I do think development, if they have the resources, and even other automation engineers they should actually try to, or have an API built to help auto-make tests quicker.
|
| Joe: | This is something a lot of tests makers may freak out about for some reason, because they're so used to end to end testing. It sounds like you're using Ruby, so what are you using to interact with that API is it just [inaudible 00:16:05] calls or how does that work?
|
| Justin: | In a way, I use just a Ruby library, like HGTP party but there's tons of them out there.
|
| Joe: | Awesome. I guess I failed to ask, how are you testing your mobile apps? What are you using, is it Appium or is it another library?
|
| Justin: | It's Appium. I'm a big Appium believer, I've been using it since, almost since it's inception. I think 2012 it was introduced at the Selenium Conference and I quickly got involved with Dan Cueller who was the original creator and then SaaS Labs and him kind-of joined forces and they created Appium.
|
| Joe: | Awesome. I've had Dan on the show, he's awesome. It sounds like almost everyone I speak to now when they say they're doing mobile testing they all say Appium. That's like the go to, it's almost like Selenium was a few years ago, it seems like it's really being the choice for testers to use for mobile testing.
|
| Justin: | Definitely, yes. Your other alternative before for iOS was UI Automation, which was fine but if you weren't a JAVA script developer it was a little challenging. Then the same for JAVA you had Robotium was your only options, but now you have Espresso and then iOS now has I believe it's like XE Test.
|
| Joe: | You find Appium is the one that seems the easiest to use or the most robust maybe?
|
| Justin: | Definitely yes, because you can write it in your own bindings. For my example I use Ruby, I can write all my Android and iOS all in Ruby. Share a lot of the same code if I wanted to, or same methods and I think it's more efficient in my opinion.
|
| Joe: | How many Nodes do you work with and how do you manage the Nodes? That's another common challenge I've heard people complain about. They're using their own Selenium grid instance. It's just management of the different Nodes that have been a problem.
|
| Justin: | Yes, that's a good question. The setup that I used, we only had a limited amount of devices. We had a good amount, internally we had like twenty different mobile devices for the company to use. At any given time a designer or another developer, QA guy or whomever wanted to use that device so I built the grid to work in a way that before any tests runs it will just grab whatever devices are connected at that point and then create the Nodes and run the tests. Then at the end of the tests tear down all the Nodes and kill the hub.
|
| Its not like your traditional sense of device lab where you have your browsers that are always connected to the hub, I spun up and created everything at run time and then I tear it all down.
|
|
| Joe: | Cool how do you do that, do you use Jenkins to kick off jobs that run something that does that, or how does that work?
|
| Justin: | Yes. I built everything in Ruby, using Rake and then essentially from that I just have a Jenkins job that will run all our Rake commands to do the build up and build down.
|
| Joe: | Awesome yes, that's a great idea. Haven't even thought of that. Some reason we use a really heavy handed tool called Electric Flow, and for me it's for CI and it's not, I think Jenkins would be better because I find a lot more information on Jenkins.
|
| Justin: | Yes. I've been using Jenkins for years, so its just sort-of my go to for CI and I'm a big fan of Travis CI for cloud as well.
|
| Joe: | Are there any books or resources you'd recommend to someone to get up to speed with mobile testing or that you've found helpful with your testing efforts?
|
| Justin: | For [inaudible 00:20:00] and device lab there's sort-of information spread everywhere. You can look, I have an example called Appium Grid in my get hub repo. You can go there. There's a ton of information on SaaS Labs and their documentation. Getting Started with Appium or Appium Examples. There's also appium.io site where again it gives you examples and how to get started as well as downloading the client libraries.
|
| Also there's a discussion board with discuss.appium.io that you can ask all the people in the community questions about mobile automation or you can use Selenium Grid and creating your own device lab.
|
|
| Additionally there's a Selenium IRC channel that's helpful as well. That's probably you'll get the sort-of core contributors of Selenium and the Selenium Grid on there. That resides on the IRC.freenode.net server. Then its #selenium for the channel.
|
|
| That's what I would recommend sort-of to get any answers out there that you want or just to communicate back and forth with the community.
|
|
| Joe: | Once again Justin, are you saying you'd rather use an in-house service or are you saying that you recommend an approach using both actually, the best of both worlds?
|
| Justin: | I do think test cloud services are the future, or are the future, and I believe you should leverage both. Our solution at Wunderlist, we would use our local mobile device lab, but we'd also push our tests out to cloud service as well occasionally to get a larger range of devices and operating systems that we didn't have locally.
|
| If you could build your framework or your test lab in a way where you can push certain tests to the cloud services, that's what I would recommend.
|
|
| Joe: | Great idea, so it's not one or the other. Are there reasons why you'd want to use one in-house and another reason why you'd want to use a cloud service? You're not saying, I'm going open source and doing it all myself, you're saying use the right tool at the right time or the right implementation for your situation right?
|
| Justin: | Exactly.
|
| Joe: | Awesome. Do you have any tips for what set of devices or service packs or versions of OS that people need to test against? A lot of times people just say test everything, but do you have a strategy used to know for your app that you always test these top five-
|
| Justin: | Yes, so there are some sources out there, some sites that you can get the market share of what are the most used devices. I believe it's always Samsung for Android. It's always the most popular, I can't remember which models, but then from those sites you can get the models and the OS versions that are the most used and you can do them by region too, you can do them by US or Europe. From that data that's what we would do, then we could take that and set the device list that we want to test against.
|
| Joe: | Okay Justin, before we go, is there one piece of actual advice you can give someone to improve their in-house Selenium Grid, Selenium mobile testing efforts and let us know the best way to find or contact you?
|
| Justin: | The one advice, yes, just don't give up. How frustrating it could be, I've been there. Just like any automation, just keep at it and you can end up getting it to do what you need it to do by working through it.
|
| I said, I've researched a lot of this, I've created my own solutions, I have them on my get hub, so if anybody wants to go there they can see the mobile grid example as well there. Then if anybody needs to contact me they can reach me at justin.ison at gmail.com
|
Good podcast with a lot of good information. I still wonder if it is ever a good idea to utilize an in house Selenium Grid solution vs utilizing a service like Sauce Labs. I understand that these cloud services have their problems. However, for a reasonable fee, you receive logs, screenshots, video, history and a bunch of other perks. To recreate this and maintain it on your own would take months of time in my opinion. Is it really worth it in the end to spend like $30K of a QA Engineers time to create a local solution? Also, personally, I could never out-Engineer an entire team at Sauce Labs by recreating their solution haha. Just doesn’t seem worth it to me.
I agree — if you can afford it — go with a service like Sauce Labs!