Welcome to Episode 94 TestTalks. In this episode, we'll discuss Agile modeling and test data management archeology with author Huw Price, a VP at CA, focusing on quality and acceleration. Discover how to treat your test cases as more of a data scientist using Agile modeling. I guarantee — you'll never look at testing the same way again.
Most test efforts are time consuming, and lack traceability as to which tests cover what application change(s). This is risky. How can you have confidence in the fact that a change that was made is actually covered by one of your existing test cases? We need a better approach in 2016. Discover one in this episode as Huw walks us through the benefits of model-based testing, and offers some awesome tips on test data management.
Listen to the Audio
In this episode, you'll discover:
- Why model based testing gets a bad wrap.
- How model based testing saves us time in the long run.
- What makes model based testing so difficult, and three ways to make it better
- Tips to improve your model based testing efforts
- What is Data Archaeology.Â
- Much, much more!
[tweet_box design=”box_2″] What is #Data Archaeology? Find out about Model Based Testing and more with ~w\@datainventor  http://www.testtalks.com/94[/tweet_box]
Join the Conversation
My favorite part of doing these podcasts is participating in the conversations they provoke. Each week, I pull out one question that I like to get your thoughts on.
This week, it is this:
Question: Do you use model based testing? Share your answer in the comments below.
Want to Test Talk?
If you have a question, comment, thought or concern, you can do so by clicking here. I'd love to hear from you.
How to Get Promoted on the Show and Increase your Kama
Subscribe to the show in iTunes and give us a rating and review. Make sure you put your real name and website in the text of the review itself. We will definitely mention you on this show.
We are also on Stitcher.com so if you prefer Stitcher, please subscribe there.
Read the Full Transcript
| Joe: | Hey, Huw. Welcome to TestTalks.
|
| Huw: | Hi, Joe. How's it going?
|
| Joe: | Awesome. It's great to finally have you on the show. Today I'd like to talk to you probably more about model based testing and also test data management, but before we get into it, can you just tell us a little bit more about yourself?
|
| Huw: | Yes, just to give my background, I was manager, at truth, Grid Tools, and we specialized in test data management and also actually a lot to do with test case optimization coverage techniques in a minimum number of test cases, maximum amount of coverage. That was purchased by CA about eight months ago and now I've transitioned into a role as kind of a VP focusing on quality and acceleration of the entire software line cycle, really, but specializing in test data and accelerated automation frameworks. That's kind of my background.
|
| Joe: | In the pre-interview, we talked a little bit and you mentioned model based testing gets a bad wrap. I know you actually showed me a really cool demo of a tool called Agile Designer that was based on model based testing. In a nutshell, could you just tell us what is model based testing and what it could help organizations with?
|
| Huw: | The reason I say it's a bad wrap, it's actually Paul Gerrard who's … I'm a big fan of Paul, we have a good time chatting, so he's the head of Euro Star and he's really into modeling. He thinks that testers should really become elite modelers as the role should be and I'm actually quoting Paul Gerrard here in that he says he got a bad press. You know there was sort of UML and the various standing standards wore out their business process modeling, et cetera and it hasn't really delivered I think. People haven't really got the benefits that they thought they would get out of it.
|
| I'll sort of give a couple of reasons why. One is that it just tends to be kind of an abstract concept. Someone creates a business process model, everyone looks at it and says that's cool, and then they go off and develop, and then things change and the model's out of date, so it's very difficult to keep it up to date. Also, beyond the visualization of being it's quite easy to see pictures, it didn't really give you too many outputs. That's why I think model based has had a bit of bad press.
|
|
| However, we were really into cause and effect modeling. That was my big thing. Richard Bender’s RBT product, I'm not sure if you're familiar with it, which is a very cool tool, but it's actually very difficult to use. You kind of need to be an electronics engineer or a high end mathematician to really get the benefit from it. What we did with Agile Designer is say, “What do people use a lot?” That's actually flow charts reality. If you look at products like Visio and products like even PowerPoint, people will draw pictures of what the process is supposed to be, so we said, “Let's take the mountain to Muhammad,” and we created Agile Designer, which if you're familiar with Visio is very easy to pick up. In other words, it looks deceptively simple, but lurking behind it is some very clever mathematics to actually calculate the total number of paths through the model. That, in effect, is your entire testing universe. That's the universe of everything you could potentially test.
|
|
| Of course, the job of the tester is to pick and that's what people do by hand in reality nowadays. They'll actually say, “Well, we're going to do a login and first thing we do is test the cancel button, so we'll click cancel. Great, that worked. Then we're going to do login and we're going to do a register, so what does register?” You build models by almost drawing your finger along a picture whereas what we'll do is we'll use some mathematics. It's a little similar to the mathematics you use when you're tracking journeys in cars. We'll calculate the optimal path through and then we'll calculate all the other paths through. What we're trying to do is make sure that every decision gate has been tested, so click login, add account, account already exists, so it either exists or it doesn't exist, so that's a decision gate and we have to test both of those.
|
|
| We convert what is a deceptively simple model into, in effect, a set of test cases. The number of potential paths could be in the billions and trillions, and we can get it down into the tens and hundreds by using this mathematics. That's really the starting point of where we began with Agile Designer.
|
|
| Joe: | What I really liked about the demo you gave me is that this is something we struggle with in my company, or my team, is that they make a change, say, to a patient search and they have 95 feature files up that run every night. For some reason they think they need to run all 95 of those features to test that patient's search. If you model it correctly, as you've shown me, you can see in reality, you'll only need to run these 5 features to actually cover this change and it was a big time saver.
|
| Huw: | There's two things on time saving there. I mean, I think the first is that the days of the requirement have died, it's all change requests, it's user stories which are, “Make this screen better.” Now, the problem with that is you actually have an existing 95 test cases, so the initial problem you've got is I now have to go through all 95 to see if they're relevant anymore. What most people do, because they're busy, is they'll just say, “Well, we'll just leave them where they are and I'll add another 5 test cases in.” Now, the efficacy of those 95 is now at risk because you haven't actually gone back through it.
|
| To go back through all 95 test cases to say, “Does this still make sense, does this still make sense,” is actually very time consuming, so what we'll do is say, “Well, we will say you've actually made this change, which affects 5 test cases. We will repair those 5 test cases and we might create you one more because you've added in a new piece of functionality.” That gives you 6. That's just the sheer labor saving device bit, but the thing that you spotted was that actually people just run regression packs regardless all the time. The whole [default 00:06:16] world now is continuous, continuous, continuous, but it needs to be focused continuous. If I've made that change to that search, the reality is I only should really be testing the 6. Let's focus on getting those 6 though and we might create a regression test, which we run in development, which we actually do to test them thoroughly.
|
|
| Also, the [correlary 00:06:43] of that is if you have a failure of a component, everyone's component ties into everything nowadays, and it's the internet of things, and all this sort of API testing, and all that sort of thing, but if you have a component that fails with maybe a regression testing that's going on on the component, you then got a problem. Do I test my entire system, which is what tends to happen at the moment, whereas actually what you should really be thinking about is, “I should really think about the 6 touch points where this API is used and I'm going to run my regression test on those 6. I fixed it, I'm now going to actually just run the regression tests on those 6 tests and I'm going to ignore the other 89 basically because they don't really touch this API in any meaningful way.” I think there's a couple of things that you picked up on there when you were looking at that.
|
|
| Joe: | I guess the thing that really disturbed me is I actually tried pitching this to one of my managers and I showed him the tool. He's like, “Yeah, but someone has to model it correctly.” I was thinking, “Well, isn't that a problem if we can't model it correctly? How do we know if we're testing it correctly?”
|
| Huw: | Absolutely, yes. You've hit it on the head. The problem is yeah, yeah, yeah, we don't have time to model because we're all too busy doing our job, but then of course what happens is your job gets too busy because you're fixing bugs because you haven't modeled. It's the classic catch-22, so it's a tipping point. You have to design test cases. The tester is modeling. They are actually building a model which should, in theory, should reflect the requirements and the requirements is a model. It comes down to who's actually going to do that modeling. In my sprint team, because the Agile Designer team is a very efficient bunch of 22-year-old developers, and we run about 3-4 week sprints, and what we'll do is say, “Look, in this sprint we need to have done the model.” I don't care who's done it. I don't care whether it's the tester, it could be me, think of me as the user, or it could be the developer, but at some stage or other we're going to have to create a pack of regression tests to test this.
|
| We need to do it to build those regression tests. Now, regression tests are very difficult to code. I mean, I think I met you at the [Lean FT 00:08:58] presentation and in fact, you had a lot of people sitting there writing code test screens. That's quite labor intensive, so the way that we think about that is why don't you just model it and we'll automatically create your Lean FT scripts. If it's going to take you 12 hours to write 5 Lean FT scripts, why don't you spend 3 hours modeling it and build 8 Lean FT scripts? What happens is you build up this library of assets, so the next time you make a change you've already got your model there.
|
|
| I think that's the idealistic approach, but what's also tending to happen is that what does exist are the test cases. You might not have the requirements, but you already have a set of existing test cases, so what we've done is said, “Why don't we just import those test cases and reverse engineer a model out of the test cases?” If you look at a login screen, a classic login, password, registration address screen which everyone uses every day, if you look at the set of test cases for that it is actually pretty easy to reverse engineer a model out of it. There's a login, it either needs some data entered into it, doesn't have spaces in it, all the usual stuff, validation checking, then you have a value that's entered, and then the next thing is does it exist, yes or no. It's a decision. If it doesn't exist, then we can move on in the next part of the process.
|
|
| What we've done is created some sort of pretty clever pattern matching stuff, which will go and try and build you a model out of your test cases. That's really the argument to go back to your manager and say you get better quality, we can reverse engineer it, but you can't carry on the way you're going because there's just too many bugs, and you're not really getting the quality, and it's slow. It's just too slow. I think that's really the argument that I use quite a lot when coming up against that objection.
|
|
| Joe: | You showed me another feature that was kind of cool, is that once you do that, you can spit it out in any format. Say you have a team that's doing [Angla JS 00:11:04], bam, you could spit that out, they could leverage it. You have another team that's doing BDD's in Cucumber, bam, you can generate it. Also, it helps re-factoring, so rather than someone going, “What? I have to change all these other things,” when you have this automatic regeneration of the test scripts, it helps simplify the re-factoring also, I think.
|
| Huw: | What we've done is if you look at a Visio diagram and it says, “Click login,” what is click login? It's in ENFT, it's a little bit of JavaScript, or it's a bit of C# script, which basically invokes the Lean FT engine to do that, so what we do is we just create a little config file, which basically maps various different parts of your automation. It's actually kind of fun setting it up. I did one the other day and all you do is you take an existing script and you literally chop it up into small chunks. You say at the very top I'm going to have an action statement, at the end I'm going to have a close action statement, so fine. That's a beginning and an end. Then you say I need to open a login page, so enter login, so that's a little piece of code. You can just drift your way down.
|
| Then what you might have a thing is there a triple box or a quadruple box which says Firefox, IE, Chrome, et cetera. That is actually built into the model that we actually need to think about when we're creating the scripts to test all the different browsers. What you could think about, and coming back to your idea about [componentizing 00:12:30] things, and this is really where the power comes in, is that you should very, very seriously think about creating these standard component libraries. If you look at something like password testing, there's positive testing and there's negative testing. The first thing is you want to make sure it rejects all the combinations of bad passwords. Now, that's actually quite a hard thing to design, but once you've done it, it needs to be just one resource that's available across the company. In fact, in my mind, it should be something that's a resource that's available across the entire testing community. We should really move towards almost like a [inaudible 00:13:06] hub for testers.
|
|
| Then you could think of the positive tests, we have a level three password, so it must contain a special character, it must contain a number, it must contain an upper case letter, so you could sort of set these different levels. Then you just expose that a common resource, which would then create you the automaton scripts or the test data to drive the automation scripts. I've been to the test managers forum in the UK a couple of times and I've pitched what would you guys think about trying to build common libraries of test assets, of test frameworks, that feed back into re-usability so that we spend our time doing way more interesting things than us all reinventing the same stuff that has been done hundreds and thousands of times across our industry. Does that make sense?
|
|
| Joe: | Absolutely makes sense and I know Jonathan Wright brought this concept up in episode 38 of TestTalk, so same type of thing. Have a hub of repository for testing assets that we just pull down. Why are we always recreating, like you said, a login that pretty much everyone that has a login should have the standard type of inputs and outputs that are needed to test that correctly.
|
| Huw: | I mean, I think the thing then is the realities that things will change because someone will miss something and then everyone in the world has got to discover that themselves, whereas if we have this community, you could almost create a little forum around some of these common tests in the banking industry things like that, credit card checking and all that stuff. If someone finds that actually there is a specific … Was it the heartbeat or the heart bleed bug that went through? We actually modeled the test cases for that that were being used and we found the bug in about five minutes.
|
| Now, if we'd had a community of people looking at the model that would never have happened, would absolutely not have happened because the industry would have come together and said, “Whoops, we're missing this. We're absolutely missing this.” I think it would just bring a degree of stability, and rigor, and reuse, and a community which seems to be lacking in the testing world really, apart from when we all get together at trade show and have a good time.
|
|
| Joe: | I'd like to switch gears now and really touch on test data management because this is something that I think I personally struggle with heavily and a lot of people I speak with also struggle with. What makes test data management so difficult for teams to handle?
|
| Huw: | First of all, people think of it incorrectly as a logistical exercise. There are products out there which copy production data. It's that dreaded, “Oh, if it's in production, that's the truth and that's all I care about.” Whereas testers and developers, our job is to be looking outside of that, looking at the edge cases, looking at thorough coverage. That's the first logical mind jump that people have to take.
|
| Then the other one is if we say focus on something like regression testing, one of the problems that we … There's a very good customer nation wide, they're pretty well organized, very well structured. We came in and we took a look at some of 1,000 test cases that they were running for a [inaudible 00:16:20] application. Now, what they found was that the automation frame, which was awesome, it was really good, was reporting bugs, but then they'd spend 2-3 weeks looking at those bugs and they were all data related. They weren't bugs.
|
|
| What was happening is that basically they were picking specific types of people or specific managers and over time the manager status had changed, which then made this particular regression test fail. Then you'd say, “Well, we need to pick a manager that's different from them.” In automation testing you've got to be very careful about preparing and picking that right test [inaudible 00:17:00]. It's almost like a pre-step is to make sure that each automation script has the right data. Now, that means the data already exists in your system and you've got to be quite systematic about going to mine that data.
|
|
| The way that we do that is we create what we call a data index, which is an abstraction of the data and with the automation framework you create this data index, you map the automation test to the characteristics of the data, not the real data, not a person ID or a manager ID. Say, “I need a manager ID who is not currently on holiday, who has an employee who is currently on a holiday. I need the manager ID and the employee ID.” Those will get allocated by our test data management system and then the automation script, all it has to do is say, “I'm test number 64A, which manager and which employee ID should I use?” That's a more structured way of doing it in terms of understanding and finding data better.
|
|
| Now, another problem that is common is with the regression test, you've got to think about a pack of data in a certain state. If you look at anything which is time based, which is most of the things we deal with in transactional systems, you can't just take a copy of a database today, move it over, freeze it in [inaudible 00:18:22], run your test today, and then run it tomorrow because tomorrow is day minus 1 now because the data has aged. The way that we tend to think about it is we would capture the data to do with maybe an account history and we pull it out of the real database. It sits outside and what we care about is the core characteristics of that data. It will be maybe an account history with 3 years of account history with something like too many withdrawals. It's got maybe more than 3 withdrawals a month, which is kind of what we're testing. We want to make sure that the penalty clause is applied.
|
|
| What we'll do is take that data then and we will push that back into the database just before we do our test. It will pivot on the dates because obviously we need today's date and we need to roll everything back, and it will assign obviously some new account numbers, some new event ID's, but we don't care about those. Those are not important to us. All we need to know is that the core characteristics of the data are correct for the test that we are running and that is a much more structured approach to being able to mine data cleverly, put data in that you know the actual expected results for and I think that is a fundamental shift.
|
|
| The third shift, which is kind of massive and a lot of this is to do with the new regulation that's coming in which is called GDPR, it's actual physical regulation now, this is not a directive you should do this, is that the use of live data in development is becoming virtually a no, no. You can explicitly withdraw consent, and I'd almost encourage listeners to do this, to just write to their banks and say, “I explicitly withdraw consent for you to use my data in your development and testing. I've heard that there is a possibility of data leaks through this route and I don't want developers and testers …” You could say, “Well, we masked the data,” but actually it's still your data, which is a legal issue, but it's one that companies are now actually starting to grapple with more seriously.
|
|
| The thing there is to think about synthetic data because production data is actually very thin. If you turn on code coverage tools, you only cover about 20% of your code with a production run, typically, whereas you should be in the 60%-80% code coverage. The easiest way to do that is to synthesize data, to create data with [parameterization 00:20:55] around it. We've got some very powerful synthetic data generation tools which will put data directly into a database. It will put it into a file, it will put it into an MQ message, but also I think, actually this is coming back to automation, is that quite often it's better to create an automation framework which puts data in through the front end of the system as well.
|
|
| Maybe you want to create 1,000 customers a night using an automation framework, but those 1,000 customers will be all different. They'll have all different types of customers, all different types of histories, anything that you consider as a test or a developer to be of interest, we would [parameterize 00:21:33] that up and actually then physically create that data for you on a regular cadence so that you have a richer pack of data, you're not spending your time hunting around for that very, very rare one, and you start using, and someone else uses it. I think it's a big subject, but I think those three things are probably a good place to start with test data as a concept.
|
|
| Joe: | Another point, I'm just trying to visualize this. I'll use a real world example. We have an old [inaudible 00:22:04] database and they have a schemer on top of it, and no one really understands what they're doing. When we're writing out automated tests, they have these sequel statements that they think they're populating the database with the right data, but they miss some tables, so they don't know really the mappings of the database behind the scenes, so we end up having corruption, and because they're doing dynamic sequel statements, can't really run the test in parallel because it causes collisions. How does this solution work? Does this sit in the middle of the database and that's what does the negotiation between the test and the actual back end database?
|
| Huw: | Yeah, there's a couple of things there. You actually bring up quite a common problem in that the technical data associated with systems is extraordinary. The good old mainframe with a green screen and a batch system has gone away. You could be dealing with 200, 300, 400, 500 systems and data is not just in a database. It might be a message layer that you're dealing with, so what we've done recently, we've really focused on this actually because it's becoming a serious problem, is that you go in and you pick the things that I care about as a developer or tester. Today I'm working on a payment system, so I care about the types of payments, the markets it's paying in, the dates, and things like that. What we'll do is we'll actually go and we create a model that sits above the data. Think of it like a mini business intelligence layer.
|
| What we'll try and do is reverse engineer rules out of the data, and in effect we'll build a model of what the data is, so you're actually paying off some of that technical [inaudible 00:23:33]. It's not perfect, but at least it gives you a fighting chance. The alternative is to dig up the assembler programmer who's now dead to try and ask them the question. Firstly, you've got to kind of try and understand that data better. Then you've got to think about how can I explore data, and one of the things that we're actually working on at the moment is that a good trick and a trick that I use an awful lot is that I will either snapshot a database or something like sequel server it's easy, you just take a backup of it. What we'll do then is we'll run some transactions through. If I was dealing with a [inaudible 00:24:10] based one, I would snapshot maybe my accounting tables or whatever the tables are interested in.
|
|
| I would then run some trades or some payments through that system and then you do a [diff 00:24:19]. That actually is a really nice way of finding out what's happening inside the database. You'll say, “Oh, right. We're posting here, we're posting there. We're doing this, we're doing that,” and all this information, this is what's actually happening in the database, not what I'm thinking. You're almost reverse engineering the [UIE 00:24:33], which is kind of a cool trick. If you're using something like Oracle there, actually you can just turn on Log Mining actually and Log Mining is beautiful. It literally gives you this lovely trace of what's happening inside the database. That's giving you some real examples of what's going on.
|
|
| Now, what you can do then is you can say, “I now know exactly what happened when I did that, so I can now create a synthetic data cannon to reflect that transaction trade,” or whatever it is, “and I'll actually create, and I'll parameterize it up, the dates, and the customers, and the ID's can all be parameterized up.” That then gives me a very powerful engine, which I can then expose through … We've got a portal called Test Data On Demand, and what that allows people to do is to actually just request the dates they want. We'll create some drop downs, fill in some information, press a button. They get an email five minutes later saying, “Here you go. Here's your account, here's a three year account history,” et cetera.
|
|
| It's not magic, but it is very systematic and there's a lovely company called [Manheim 00:25:35]. I met them and they came up with the term Data Archaeology. You have to spend your time actually hunting around and actually understand the data that you're dealing with before you can think about improving it. I said, “Well, how long does that take you?” They said, “We normally spend about two weeks on a system that we've never seen before going through and building, paying off that debt. Once we got that out of the way, we then become extremely productive and then in the next week we will create a ton of data assets, which then we expose to the developers and testers that they can push that information directly out, then.”
|
|
| I think that is quite a structured way of being able to do that. There's more to it than that, but I mean hopefully this is kind of giving you a flavor of some of the approaches that are quite effective.
|
|
| Joe: | Okay, Huw. Before we go, is there one piece of actual advice you can give someone to improve their model based testing or test management system?
|
| Huw: | You got to start. You just got to do it. People put things off. Download Agile Designer, model something, measure your coverage of resisting test cases, go show your manager, and say, “We're either doing a brilliant job, [inaudible 00:26:47] case fantastic, but it's taking too long,” or, “We're not doing a good enough job.” The only way to do it is to actually do it. People procrastinate a bit, and it's fun, and it's a different way of thinking, and I think that's probably the thing. If they could do that, that'd be great.
|
Comments are closed.