Leonardo Lanni
Yeah, yeah, that's the right way. Mutation testing, for me, the first time I heard that, that was a great idea. Let's simplify with an example. You have your code and you have some, let's say, unit tests, which is covering the code that you're building. The main problem is that, or the challenge I would say is that, how do we make sure that those unit tests are really effective? A lot of times people talk about coverage as a good metric to evaluate unit testing, but covering all the code is definitely a good starting point. But is that really enough? My personal opinion is no, because you could write just as a matter of proving that is not enough dumb test cases that just asserting that true equals true or absurding the obvious. Test coverage, covering all your code through unit test is a great idea, but it's not enough. There is a need for me to understand are those test cases built in a way that if there is a bug due to some code modifications, they will be able to catch the bugs. The need is measuring and verifying the quality of those test cases. Are they valid? They really test what is important, what then whatever that fails could break things and could fail to deliver business value so they could put the entire application under risk. The mutation test comes in a very cool idea. So the idea would be, what if we have a programmatic, let's say automated way to modify our target test code? Imagine there is a tool that takes an input, the code that we're testing, that we are producing, our application code. Modifies it, mutates it, if we want to use the right terminology. What does that mean? It basically means modifying, for instance, the condition in an if-then flow. If then flow is executed with a major sign, you just flip it and then you create a modified, a mutated version of this code and then, you execute all your test cases against this mutated version of your code. So why would you do something that's, the first glance you may say a little bit, that's a way, why would they do something like that? Well, the idea is very smart actually. If you run a test case against a modified version of your code, what would be the expectation? The expectation would be that your test is failing because by modifying the code, and by having the test case still testing the original version, you would expect this mutation to make the test cases fail. Now, this is something that I would say applies in general. We could always find cases where this doesn't apply, but in general, mutating the code and running the automated unit tests against this code should make the tests fail. The mutation testing tools, all they do is they take your code, they apply some mutations and mutating the code could be done in so many different ways. So also the challenge is finding the right way to mutate the code. And this is also a computational challenge because if you imagine there could be billion of ways of mutating the code, you want to pick the right ones because at the end of the day, this mutation testing is something that you normally would add on top of your testing pipeline. And this something that must have a time budget. Mutation testing would take 20 more hours, it's not worth it because nobody can wait extra 20 hours to get test results. But imagine this can be done in 5 minutes, and at the end of this automated activity, you have a report that would tell you, hey, hold on, because there are a couple of test cases that are not failing against this mutated code. All you have to do is say, hmm, that sounds a little bit weird, so you just go and analyze those test cases manually. And you try and understand why they didn't fail. Now, there are two options. The option A, is in general that we can say there is a reason why they're not failing and then life is good. Then you have validated your test cases to be good. But the worst case scenario is, ha ha, you may have found a test case that it's not doing a right job. So basically it's asserting something that it is meaningless, there could be an error, there could be something wrong in a way that if you mutate the code, your test case is still passing. And honestly, Joe, this is the worst situation ever because we all rely on our test cases. We all assume that whoever wrote those test cases, and they could have been written many, many years ago. They were reviewed, they're correct, and life is good. And what is even worse, technical debt is a really big anomaly in companies. Do we really have the time to go and review the test cases? And do we really want to review test cases which are passing? And this goes a little bit into psychology. When I first, if I see green stuff on my screen, happy days, you feel like popping the champagne and celebrating. The little thing that says, hey, are you sure those test cases are right? Maybe they're passing, but they're not doing the right thing. I may say, It would be great to validate that, but it's effort, right? But if I had a tool that would do that automatically and would point out to me a couple of test cases of maybe hundreds that could be suspicious, happy days, it's well-spent time because if you find the wrong test cases, you fix them, then you have increased the quality of your test automation and you sleep much better in the night knowing that your test cases are really effective, they're delivering quality.