I recently interviewed Amber Race, an SDET at Big Fish Games, about how she uses automation to test video games.
It turns out that most of the testing she needs to perform is a good fit for automation are back-end, API-type tests, and her tool of choice is Postman chrome.
What is Postman Chrome?
Postman Chrome is a rest client that started off as a potman chrome extension browser plugin but recently came out with native versions for both Mac and Windows.
At a high level, you can use it to send a post request to your web server and it gives you the response back. It allows you to set up all the headers and cookies your API expects, then check the response when it comes back.
That's its basic functionality, but it also has lots of other features built into it like excellent cookie management that can sometimes be hard to manage with other API tools.
Postman chrome plugin and app supports every HTTP method you can think of — including some you might even know about.
For API validation, it has basic checking you can do when you receive the response. A common example of this is when you send a request to an API and get a response back, you verify that it returns a HTTP 200 OK status, or you make sure that the response contains a certain string in it.
It also has a postman collections which is an executable API description feature that you should check out.
Exploratory API Testing Using Postman
When most folks think of automation they don’t necessarily think of using test automation tools, but that’s a mistake.
In Amber’s experience, when she is performing back-end testing she finds herself using Postman more and more to assist her in exploring her team’s APIs. It really helps her to learn what an API does.
The main way she does this is by tweaking the API request to see what comes back. It's a really useful tool for both automated and manual testing.
She explained that she is often given a scenario — load testing a web server, for instance. In situations like this, you don’t always know a whole lot about what the web server is supposed to do; you just expect to load test it without any context.
To understand how to create a realistic test, you need to really look into the game to see how it's communicating with the server, then try to replicate those calls using a tool like Postman. From there it’s just a matter of seeing what's there and determining what works and what doesn't.
Amber also mentioned that in her experience with automation, 95% of the bug finding happens when she’s writing the automation, then it just runs. Occasionally it'll come with a regression bug.
The exciting and interesting part is the writing of the automation when you're exploring the API.
[tweet_box design=”default”]Play around with your API. Once you get a test figured out tweak it, experiment with the API is really an important thing @ambertests[/tweet_box]
Do Postman Scripts Run in CI?
Because Postman’s API tests are basically tests written in JavaScript, they’re easy to integrate into your continuous integration environments.
There's a collection of runners for both a GUI and command line runner options. It also creates some reports for you that can be outputted to XML in a way that a tool like Jenkins could display the results.
Using Postman to Create a Reusable API Framework
So how is Amber successful with her API testing?
Firstly, it helps to have her unified API development scheme defined. For each of the services at Big Fish Games, they have a core set of services that most games are calling in the background.
As a result, she can have a core set of tests that test those services, and all the other testing builds on those.
Amber feels that this reusable, component-based API service approach is really for API testing.
Postman is even useful at this level. With Postman, Amber’s team uses it to research and figure out all the features of their APIs. They then take those findings and post them on their team’s WIKI. This helps to ensure everyone on the team understands their APIs, and that everyone can run it and make certain that what they're doing doesn't break what's already there.
Postman makes it pretty easy because it's a very collaborative tool. You can even put a button on your internal website saying ‘run in Postman’ to kick of a script.
Amber’s Actionable Postman API Advice
The best advice Amber feels she best advice that can offer to Postman Chrome automation engineers is to just start using it to play around with your APIs.
Once you get a request-response conversation figured out, just tweak the request, tweak the headers and tweak the cookies (if there are any cookies involved).
It really is all experimenting; playing with the API is actually a pretty important thing.
Install Postman Chrome
Not only is Postman is easy to use its easy to install and get started with as well. All you need to do for the postman Google chrome install is go to the chrome web store and add it to chrome.
For the Postman chrome app, you can download it from the getpostman site.
It runs on Mac, Windows, and Linux.
More Postman API Testing Awesomeness
| Joe: | Hey Amber. Welcome to TestTalks. |
| Amber: | Hi. Thank you for having me. |
| Joe: | It's awesome to have you on the show. Today I'd like to talk about your experience at the gaming company Big Fish Games and also at Microsoft as well as testing backend services using things like Postman. Before we get into it could you just tell us a bit more about yourself? |
| Amber: | I've been a software developer in test for 15 years. I started at Microsoft, started as a contract tester and moved into more technical code-related testing. Since 2008 I've been working at Big Fish Games doing various server side testing mostly, and also a little bit of dabbling on client side testing as well. |
| Joe: | Very cool. I guess the first thing that comes to mind is can you contrast what the differences may have been between your automation experience and Microsoft versus a gaming company like Big Fish? |
| Amber: | The biggest difference when I switched in 2008 was that at Microsoft it's a very enclosed technological environment. You're using Microsoft visual studio and Microsoft C# and everything Microsoft. When I went to Big Fish Games it was an open source environment. I had to relearn all the tools, learn Eclipse and Java and JMeter and all this stuff that I hadn't previously had any experience with. That was a major difference right there. |
| When I was working at Microsoft I was working in the Windows organization which has a very slow release cycle. I think I was on the Vista project for probably 5 years, as an example. That was a big difference then moving to Big Fish Games even though this was before mobile, even the website had a very quick turnaround. Things would happen in a matter of 1 or 2 months instead of 1 or 2 years. That was a big difference as well. | |
| Joe: | Are there any transferable skills you found help you to adapt to an environment that's not so one sided, where there are different technologies that you have to learn? I know I guess you probably knew C# so you may have had to learn Java. Any tips for someone that's transferring skills? |
| Amber: | I did know C# and Java's very similar to C#. In that sense it was not so bad. I also started using Python [inaudible 00:02:07] which I never had coded in Python before. Program languages are very similar in their logic. That's the important thing. Liberal use of stack overflow, of course, is very helpful for any sort of coding exercise. I think just being open and just thinking just cause I've never coded anything in Ruby or never understood anything in Ruby, if you understand the logical flow of a programming language pretty much any language you can at least get some familiarity with I think. |
| Joe: | I definitely agree. I also was thinking that probably in Microsoft, maybe this is just a generalization, you probably were doing more UI-based automation before a gaming company, I just assume, because it's so hard to automate the front end there's probably a lot of backend EPI-type testing. Is that true? |
| Amber: | That's correct. |
| Joe: | Yeah? |
| Amber: | At Microsoft I actually was in a little bit of a different niche section. I was working in the handwriting recognition group. I did a lot of work with data sets for framing and testing the neuro network programs that were doing the handwriting recognition. I also did quite a bit of API level testing for the NKPI for tablet PC. I tried to stay away from UI automation even at Microsoft, although they did, Office especially has a lot of very useful tools for UI automation, which I did use to some extent. |
| In games it's very difficult to automate. Game play it's generally not worth the trouble to automate it. A human tester can judge the quality of game play much better then a computer can. | |
| Joe: | I guess that's a great point. How does automation work in your company now at Big Fish when you're testing a game? I think you do a lot more backend testing, but is there a process you work through? Do you have live tester actually test it and then you automate things in the background? How does that work? |
| Amber: | Usually how it works is the game play is happening. There's the gaming engine and then if they're hooking it in with backend services the development cycles are not all that closely linked together so game play development can be happening independently of the server side. They'll come up with this game and they're playing the game and they're like, “Oh, hey, we want to add chat to our game,” or “We want to add gifting to our game,” so now we're going to pull in this backend service that there needs to be tested. Ideally everything would come together at the same time, but that's not how it generally works in practice. |
| Joe: | When that's happening how do you get pulled into it? Do you work on a sprint team or does someone say, “Look, we're going to add chat. Can you test this for us or automate some of this for us?” |
| Amber: | For the past couple of years I've worked on a ninja-like engineering team that will come in and integrate preexisting games with fixed services and put them together. We're brought in like mercenary types to come in and make this happen, then we move onto the next game is what I've been doing for the past couple of years at least. |
| Joe: | What kind of backend services do you typically test? Are there APIs at all games use that are common between them? Maybe you have a movement engine or something? How does that work? |
| Amber: | The constant dream is to have a single game engine that all the engines use that all the games use, and then you could test that engine. What's happened a lot at Big Fish Games is there's a bunch of little studios put together and everybody is developing their own game and it can be difficult to get everybody working in the same engine or working even in the same programming language. This game is going to be using coco's and this other game is going to be using unity and then this other game is using something completely different, so that's a challenge. |
| What we have tried to do is have a lot of our own backend services. We have an SDK so that various games, even though they're written in all sorts of different languages and platforms can then plug into our backend services. | |
| Joe: | It sounds awesome in a sense that it seems like you're probably learning all the time that throwing you on a project that maybe uses a technology you're not used to and all of a sudden you're like, “Wow, this is cool. How do I do this?” |
| Amber: | Yes. As far as testing game engines themselves, that's not an area that I have a lot of expertise in, although I think I'll be learning more about it in the future. As far as with backend services it can even understand we're just plugging one thing into the other. It often gets very messy where then you have to have get into the game to plug it in, make sure everything is working. The relationship between the client and the server can get very tight. |
| Joe: | It sounds like, it doesn't sound like you are, given a session, a STPCON? |
| Amber: | Yes. |
| Joe: | I believe your session's on Postman? |
| Amber: | Yes. |
| Joe: | I guess at high level what is Postman? |
| Amber: | Postman is a rest client basically. You can use it, it started as a Chrome plugin. Now there's native versions for both Mac and Windows. The simplest level you can use it to send a post request to your web server and then it gives you the response back. You can setup all the headers and all the cookies and everything the way you want it and then check the response when it comes back. That's basically what it is, but it has a lot of other features built into it. The cookie management is really good. |
| It supports every single HT method you could think of including some I don't even know what they are. It has basic checking that you can do when you get the response. You make sure that it's a 200 or you make sure that it has a certain string in it. A lot of this is backend so you can do very simple API checking with it or you can use it for exploring APIs. You can tweak the request and see what comes back. It's a really useful tool for both manual and automated testing. | |
| Joe: | Very cool. You just brought up something that I think is a big, big point. That is you just mentioned exploratory testing via using a tool. Some folks, I think, think of automation as just a mindless script that goes in and it runs by itself. Sometimes you can use tools to get you to a certain point to then do minimal testing. Is that how you're using Postman in that regard? |
| Amber: | Absolutely. I've been using it mostly just to learn what an API does. For example, I'll be given a test to load test this web server. What does the web server do? We don't know. Just load test it. It's a matter of looking into the game and seeing how it's communicating with the server and then trying to replicate those calls using a tool like Postman and then just seeing what's there and what works and what doesn't. My experience with automation is that 95% of the bug finding happens when you're writing the automation, then it just runs. Occasionally it'll come with a regression bug. The exciting, interesting part is the writing the automation when you're exploring the API. |
| Joe: | Awesome. I think that's another great point. When you do trader automation script on Postman can it run by itself? Can you run in CI? How does that work? |
| Amber: | It's basically Java script that it uses to do that test, so it's pretty simple. There's a collection runner both a gooey runner and there's also a command line runner. You couldn't run it as part of a Jenkins continuous integration environment from the command line. |
| Joe: | Is there a reason why when you use another tool sale like SoapUI to do, is it just you happen to find this tool and it happens to work in your workflow? How did that come about? |
| Amber: | We don't use SoapUI at Big Fish Games, so I haven't had much cause to do testing in that. This tool I just happened across it. It's super useful as a client. It has a very rich interface that a lot of the previous rest clients didn't have. It makes it real easy to share your knowledge with your coworkers because you can package up all your requests and expected responses and you can send it off to somebody else so that they can look at it also. It's just a really useful tool. |
| Joe: | That sounds cool. I didn't know it had that functionality. Does it create reports also? |
| Amber: | It does create some reports or you can output to XML in a way that something like Jenkins could display the results. |
| Joe: | Very cool. I also know that sometimes with certain tools, I'm just thinking of UFTAPI for some reason, it has prompts handing different types of authentication. How does Postman handle it? |
| Amber: | Postman has a lot of authentication. It handles a lot of different kinds of authentication. It handles basic authentication, [inaudible 00:10:40] 1 and 2, Hawk, and I guess I could look at it to refresh my memory. Just out of the box it handles all of that stuff in a way that makes it easy to test the authentication or include it in your request. |
| Joe: | I think most people are familiar with UI-based automation frameworks, but I do get a lot of questions sometimes how do I create an EPI-based automation framework. Do you have any tips of tricks for how someone can create a reusable API backend-based automation framework? |
| Amber: | It helps a lot if you have a unified API development scheme as well. For our services at Big Fish Games we have sort of a core set of services that everybody is calling so we can have a core set of tests that tests those services and then all the other testing build on those. |
| There's a service that checks the player's identity. That's the core of any service that's going to do anything else with the player's information so we have this identity-type service and then we can have tests that work on that service. | |
| I think that something that can be really useful for API testing, if you want to have something that is portable to other things, something I've actually found useful is using the Junit rules. If people are familiar with Junit they have these class rules where you can start up a service in just one line of a rule. If you have this service that is common to all the other services you can make a rule that says, “Now I'm going to throw up this identity service and that's what I'm going to be talking about at [inaudible 00:12:18].” | |
| I think with Postman what you do is one person is doing the research and figuring out all this stuff, and then you're saying, “This is our API.” You put it on the wiki, this is our API, and everybody can run it and make sure that what they're doing doesn't break what's already there. It just makes it pretty easy because it's very collaborative tool. You can even put a button on your internal website saying run in Postman. Run my API's in Postman and make sure everything works. | |
| Joe: | You're testing APIs. This is something I struggle with and I'm curious to know how you handle it there. A lot of times we're doing UI tests because we don't have an API. I've been trying to tell developers if you create an API the test be faster, quicker, probably more reliable. How do you get developers to create APIs for you to test or is that just part of the way development is done there at Big Fish? |
| Amber: | I think at Big Fish generally people are good about using APIs. We work with a lot of different game developer companies publishing their games. We do provide them with an STK so they can develop against our APIs. That's where we're stepping forward with the API and saying, “If you want us to use your game you need to use these APIs.” That helps a lot. |
| I think that people generally are seeing that if there's an API that is happening on the backend, then when you're testing that API if something goes wrong you don't have to wonder is it happening on the server, is it happening on the server, is it happening on the [inaudible 00:13:47] when you know that service is working is correctly so you can concentrated on what's happening in the UI, for example. | |
| It's very helpful that all the levels of development process to have a stable API, a stable backend service that people can count on as a foundation for the rest of the game. | |
| Joe: | For gaming companies I'm not sure if there any third party applications that your games rely on, and how do go about testing them? |
| Amber: | Yes. There's a huge ecosystem of third party vendors for applications to track player activity, to track whether they're clicking on an ad, to track everything, which is great for smaller game developers. They don't have to worry about figuring all this stuff out. They can contract with Upside or Contagient or all these other vendors that's going to do this work for them and it's really good. |
| Big Fish also uses a lot of these vendors. We also try to do more in house because then we have more control over our data and also we don't have to pay a third party vendor. There's a lot of third party solutions out there for these issues. | |
| Joe: | Once again, it seems a lot of this now is mobile play, so a lot of mobile devices. How do you handle mobile testing? |
| Amber: | When I started at Big Fish mobile was not a thing at all. It was all about PC and PC gaming. Now it's all about the mobile. It makes it a lot more difficult, especially for UI testing. For the backend it's pretty much the same, which is one of the reasons I like backend testing. I don't have to worry if it's Mac or if it's Android or if it's a Galaxy or if it's a LG or whatever kind of lollipop, marshmallow, IOS. I don't have to worry about any of that on the backend. Of course when you're testing the front end you have to worry about all that stuff. We have very large game QA team with all sorts of different devices to test to cover all of that. |
| Joe: | Are you sitting around with [inaudible 00:15:53] on your heads playing the games? |
| Amber: | We actually are starting to look into virtual reality, although I don't know a whole lot about what's going on with there. Yes, that is the next big thing. |
| Joe: | You guys hiring? |
| Amber: | We are hiring. Go to bigfishgames.com jobs, we're hiring. |
| Joe: | Awesome. |
| Amber: | The mobile aspect is it has a lot of competition, but it's a lot of fun too. It's really expanded the gaming universe considerably. |
| Joe: | Very cool. You mentioned backend testing a lot. I'm just thinking API, but are there other kinds of testing or tools you use for backend testing? What is the backend that we're talking about that you're testing? |
| Amber: | When I'm talking about backend usually we're using Java services. We also use PHP services, sort of your basic lamp stack, the old school lamp stack is still a thing. In addition to testing APIs we do a lot of load testing to make sure that our services aren't going to fall over when a game gets big, which we hope they all do. That's also an important part of what I'm testing at Big Fish. |
| Joe: | You mentioned JMeter. How much do you use JMeter and how do you use it for simulating users playing the game or just a service that maybe consumed by a particular game? |
| Amber: | I use it in a couple of ways. When I'm doing general load testing I do try to simulate what a player's session is going to do, so I do a lot of research into what happens when you're actually playing the game and what falls or how many of each call is being made during the session, how many, how long is a standard session, all that sort of stuff. All that goes into making a general load test and then JMeter can then throw all this in and see how many requests we get for regular session play. I do try to mimic actual session play to that extent. |
| Also it's good for singling out certain calls that maybe have especially glaring performance issues are especially key to the game experience, so singling those out as well in JMeter. | |
| Joe: | Is there one thing you see over and over again in your experience that most people doing API testing are either doing wrong or they just don't know that they can do something that would make it better? |
| Amber: | I think that there's, as with all testing, it's important to not just test what works, but also to test what doesn't work. You can never, ever, ever trust a client to send to the server the correct information, never. Either because of bugs within the client or people trying to be malicious or any sort of thing. You have to test for null. You have to test for empty screens. You have to test for if something is in the database or not in the database. There's a whole huge area of failure testing which is really important for APIs especially. You have to decide if the APIs are going to fail they're going to get something at some point that they don't understand. What are they going to do? Are they going to crash the whole game or is everything going to just move along? |
| These sorts of issues are really important. It's not enough to say, “Oh well, I made this ball and it works.” You have to figure out all the ways that it could not work and make it work. I found that it's very common for people to develop an API and not account for if this string is null, or if this perimeter is just completely missing from the call. | |
| Joe: | I definitely agree. I think a lot of people just focus on the happy pat. I think one reason for that is sometimes they don't know how to emulate those edge cases. How do you emulate a server down? Do you mock things? Do you stub things? |
| Amber: | That's where a tool like Postman can be real helpful for looking out for these edge cases. You can easily take their request to be whatever you want, if it has something that's missing or it has something that's in Chinese instead of [inaudible 00:19:43] characters, you can do all that stuff in a tool like Postman. It'll be clear right away if the service can handle it or not. |
| As far as server down scenarios, proxies are, that's more of a client side test. It depends on which service. If the service is talking to another service then sometimes with mocking you can pretend those other services are down. | |
| Joe: | You did mention Chinese. Did you do a lot of localization-type testing? |
| Amber: | Just to the extent the usual issue for localization has been if the localized strings are being stored and retrieved correctly from the database. This is something that we've been burned on several times where the collation and the sequel cable is not correct so you put in Japanese and you get question marks back. That kind of thing. Even if the service level, a little bit of non-asking testing is important. |
| Joe: | Are there any books of resources you recommend to someone to get them up to speed on API testing that you found helpful? |
| Amber: | Besides stack overflow? I have to admit that I haven't read a whole lot of books. Things like the classics like James Box, Contact Space Testing, those are things that every tester should read I think. As far as a specific book for API testing I can't think of one specifically. It's all I've just learned as I went along. |
| I think it's important for people not to be intimidated that they're working with something that isn't the UI. You're asking the service something and then you're getting an answer back. You can phrase your question to the service however you like. You have a lot of freedom when you're testing and API. | |
| Joe: | They're great advice. I definitely agree with that. For some reason people get freaked out when they have to go under the covers. To me it's almost easier. |
| Amber: | I have done a little bit of play testing. I definitely prefer testing behind the scenes instead of testing natural game play. I find the behind the scenes more enjoyable personally. |
| Joe: | Amber, before we go is there one piece of actual advice you can give someone to improve their API Postman testing efforts and let us know the best way to find our contact too? |
| Amber: | The best advice I would have is to just play around with the API. Once you get a request response conversation figured out just tweak this, tweak the request, tweak the headers, tweak the cookies if there's cookies involved. I think that's just experimenting, play with the API is really an important thing. |
| As far as contacting me, I'm on Twitter at ambertests. You can also email me, amber.race at outlook com. |
Which is better rest-assured or postman ?
I tried both a little for a few hours. It seems that postman would be better because you don’t have to write the given when then and all the boilerplate (content type json etc) that goes with rest assured. If you can run postman scripts in a CI, then that would be great too.
I’m a big fan of Postman.
It lets you chain API calls and cascade data from a response to the next call using variables. You can also populate those variables using an external data source like a spreadsheet, and even create your own variable values using little snippets of JavaScript code.
You can also integrate with Jenkins using Newman, and in 4 years of using it I have found Postman to be 100% reliable.
postman is completely useless, espically the native version, it never posts to the url that i tell it to , and software is unsupported, waste of time