Contract Testing in Action (Book Review) with Marie Cruz and Lewis Prescott

[00:00:00] In a land of testers, far and wide they journeyed. Seeking answers, seeking skills, seeking a better way. Through the hills they wandered, through treacherous terrain. But then they heard a tale, a podcast they had to obey. Oh, the Test Guild Automation Testing podcast. Guiding testers with automation awesomeness. From ancient realms to modern days, they lead the way. Oh, the Test Guild Automation Testing podcast. With lutes and lyres, the bards began their song. A tune of knowledge, a melody of code. Through the air it spread, like wildfire through the land. Guiding testers, showing them the secrets to behold. Oh, the Test Guild Automation Testing podcast. Guiding testers with automation awesomeness. From ancient realms to modern days, they lead the way. Oh, the Test Guild Automation Testing podcast. Oh, the Test Guild Automation Testing podcast. With lutes and lyres, the bards began their song. A tune of knowledge, a melody of code. Through the air it spread, like wildfire through the land. Guiding testers, showing them the secrets to behold.

[00:01:53] Hey, do you want to learn more about contract testing? If so, you're in the right place because today, we're going to be talking with two awesome experts on all things automation, but more especially contract testing today with Lewis Prescott and Marie Cruz to talk all about. They have a brand new book. I think it's still in the works, but you could still get I think, an early version of it called Contract Testing in Action. Really excited about this. If you don't know, Lewis has been on the show before, he is a test specialist at IBM, has over nine years of experience in software testing, he is a recognized champion in contract testing and a course author at Test Automation University as well, and a totally crazy active member and mentor in the tester community. And we also have with us the one and only Marie Cruz, who is a software test with over ten years of experience. Currently, she's a senior developer advocate at Grafana Labs, and she's previously worked as an engineering manager responsible for driving continuous testing and quickly improving many improvements across many organizations as a Principal Engineer focused on introducing recommended best practices for testing and test automation frameworks, one of them being around contract testing. Really excited about this episode? You don't want to miss it. Check it out.

[00:03:00] Hey, Lewis and Marie, welcome back to The Guild.

[00:03:04] Marie Cruz Hey, Lewis. Hey, Joe.

[00:03:07] Lewis Prescott Hey.

[00:03:07] Joe Colantonio Hey.

[00:03:09] Marie Cruz Yeah, really great to be here.

[00:03:12] Speaker 2 Yeah. Awesome to have you both here. It's been a while. So you two, like I said, are really, really busy. And I'm always curious. I always authors this is Why? Why write a book when I know you two are crazy busy? Marie, let's start with you. Why? Any thoughts? Why write a book at this time?

[00:03:27] Marie Cruz Yeah, it's always in my bucket list to write the book in terms of professional. And I reach out to Lewis over a year ago, because I know Lewis is really passionate about contract testing. And we both noticed that while there are a lot of blog posts, Pact documentation is great, but there's no dedicated book about contract testing. And I think during this time as well, I was inspired by Mark Winteringham's dedication in writing the book about testing web APIs. So I reach out to Lewis. I asked if he would be comfortable writing a book with me because it's always, I guess, easier to do it with someone that you know from the community because taking it on just by yourself, it's going to be a mammoth work. So yeah, it basically came from that. And about the year after the book is here. So it's been a long hard work, but I'm glad that we both did it.

[00:04:33] Joe Colantonio Absolutely awesome. Like I said, it is a resource that the community really needs. So Lewis, you've been talking about contract testing forever. I think the last time we had you on the show, you talked about contract testing. Very passionate about contract testing. So why a book now on contract testing?

[00:04:47] Lewis Prescott Yeah. As Marie mentioned, her fault, she got me involved in the book. No, I think, as Marie said, like there's no dedicated book. And like one of the biggest hurdles is getting started with contract testing. So like we really wanted to address that with the book. But yeah, I think it'll be a long time before I put pen to paper again.

[00:05:13] Joe Colantonio Yes. So talking about it's going to be a long time. I notice it's in, I don't know if people know about Manning. They have like, a an early release program, it sounds like. And it looks like there's 4 chapters out of, I think, 12. So all those chapters are already done. And I know this is like a real publisher. It's not like one of my books that is self-published is a real publisher. It's already in the works and you're just waiting for them to be looked over and make sure that blast before they go live. How does that work?

[00:05:39] Marie Cruz Yeah, pretty much. Lewis and I have been really working hard delivering the rest of the chapters. We still have one more chapter to finish, which is around the bi directional contract testing and another round of reviews as well. But I think what Manning does is they try to regularly provide updates to the user. So if people are going to purchase it as part of their early access program, the goal is to deliver each chapter every month or some new updates. But yeah, most of the chapters are done. We've had amazing reviews as well from external people from Manning side as well as well as from the community. So we've incorporated most of those, and I think we have another round of final review. So fingers crossed that yeah, it's going to be really, really out soon.

[00:06:33] Joe Colantonio Awesome. We're going to have a link to it in the show notes down below. I know I've been seeing a lot of post on, on social media, and it looks like it's getting a lot of reviews already on just early, chapters. So really excited for people to grab the full copy. But you can get an early copy now in the links down below, depending on when you're listening to or the full version as well. So I guess let's kick it off by people if they're just listening and they're like, oh, this is great. What's contract testing? Let's start with principle one, Lewis. So how would you explain what is contract testing before we dive into more of the book?

[00:07:01] Lewis Prescott Yeah. We really like delve into this in the first few chapters of the book. So it's really trying to outline what the different components of it are. So who is a consumer? Who is the provider? Who are the interactions between? Where do you put the contracts once you've generated them? How are they different to like your open API specifications? They really kind of delve down into those myths, into those preconceptions of what it is. We delve into those in the introduction, and the way that we explain it is through, like that mental model of, okay, this is a scenario that you have in your application, which is, for an example, an API. You have an API which communicates with a mobile app or web app, and that forms your contract, right? The two things communicating together. The contract sits in between. And then you've got all of the nuances of updating that contract to I need to make a change to the web app. How am I going to make that change with this API? Is that going to break it? How is it going to behave? In a short stuff it's about that communication right between two systems. That doesn't have to be an API. That could be an event or a GraphQL interface, but it's that communication between the two systems.

[00:08:34] Joe Colantonio Awesome. And Marie, why is contract testing important, do you think nowadays to write a book full book on it? So is it something you've been seeing? I know you said there's not a lot of resources out there, but are you seeing it's a technique that a lot of people are missing out on.

[00:08:46] Marie Cruz I definitely think so because I think nowadays as well, there's a lot of focus, especially with microservices. And as we increase the number of APIs that we have to consume, how do we then test those different integration points? Because while that type of approach is really scaling well, I think the integration tests that we know currently, which is we make a Get request to a specific endpoint and then verify if all the data, if everything there is correct. That might not scale as well. Let's say if you have more than one or if you're like Netflix or Amazon who have loads of different microservices, how do you then ensure that any small changes that each team deploys to production doesn't break any of dependencies out there because a common scenario that I've seen in the past was we have a consumer that could be a web app, and then we have a data provider. Both of them will have their own set of tests. The provider might deploy some changes from their side, and then the test that they have in their pipeline that are running. But it's quite common to miss the sort of scenario where have they communicated it to the consumer? Because most of the time we end up with the changes being deployed to an environment already and not knowing that that has broken some sort of integration that the other team is using. So I think nowadays it's a type of testing that both Lewis and I believe has that great impact, because it's really gonna let you know way, way earlier on if any of your changes are breaking any of the expectations from your consumer, or it just gives you that early confidence that whatever changes you're about to make, it's not gonna impact any other systems out there. Whereas with traditional integration tests, you typically have to deploy that change first because you test with actual test environments and you end up, I guess, having a longer feedback loop, because then you've noticed that there is a bug and then your tests are failing, and then you then have to communicate to that other team. Whereas with contract testing it detects that issue earlier on. So before even deploying those changes to an environment.

[00:11:22] Joe Colantonio Awesome. So let's dive into the book a little bit more. It's broken up into three parts. The first one is I think, introduction to contract testing. So if anyone's starting from scratch this is obviously where you start. Everyone starts. Part two is consumer driven contract testing and the third is provider driven contract testing. So in the first part, I think we went over the why the how to give a technical overview in chapter two. Do you have like a how do you give a pretty good example there? Marie, any other examples of the contract testing principles as they applied to maybe a real world scenario?

[00:11:54] Marie Cruz Yeah, I think in chapter one, I've definitely provided an example where I think most of us have encountered where imagine you're working on the Friday, and I think it's good to deploy changes on a Friday, but sometimes teams are still scared to do it. But the teams who want to deploy on the Friday, they have their tests running on the pipeline, but then suddenly there's now some sort of alert or notification that something's been broken in production, and that has happened quite a few times. So the way we've introduced contract testing is imagine if there is a way for you to even catch all those changes before deploying it to an environment. Another way of explaining it is using the contract example that we have with Manning. So I've explained it in a way that contract testing is like when you sign a contract with someone, you have to make sure first that everyone agrees with what's written in the document, whether that's from the consumer's view or that's from something that the provider has to provide to the consumer so that there's no surprise element right at the end. In this scenario, though, the one that's driving it is the consumer. So the consumer will say, okay, if I make a Get request to this particular endpoint, I expect to receive the status code, this different data that has this particular data type and so on. And in that contract as well, it will define like what sort of like other scenarios that are maybe quite difficult to emulate in an integration test because you need to, for example, emulate. Okay, what data do I need to make sure that I receive a 404 code? Or what scenario can trigger if it's going to be like a 403 unauthorized? All of those will be documented in this contract. And then with this approach, the provider can then say, oh, okay, this is what my consumer wants. I don't need to over engineer my API. So it's helping the provider in a way as well, so that any changes that they work on is really what the consumer needs. So that's the focus of the whole consumer driven approach. Yeah.

[00:14:20] Joe Colantonio Nice. Lewis, any favorite points in the first maybe three chapters that you want to highlight or you think maybe people struggle with that this book addresses?

[00:14:29] Lewis Prescott Yeah. So I think one of the things that we really like touch upon is how you would sell this. How do you get your team bought into the whole contract testing process? Because I think lots of people get stalled at that point because you do the proof of concept, you're like, oh yeah, we can definitely start to use this. And then you go down the route of, okay, we need to deploy an instance of the broker. Why would we do that? We've already got integration tests which test the scenario. Weighing up those decisions of why would you choose this approach over another approach. And then yeah, providing those benefits of that early feedback, being able to debug your API issues quickly using this tool like so, providing those benefits so you can sell that back to your team and have a strong argument.

[00:15:24] Joe Colantonio Love it. And so this is the first part of the book. Second part of the book. I think it's the meat of the book because I think it has like seven chapters, consumer driven context testing. You covered things like web application, mobile client, provider contract test, GraphSQL, event driven architecture, storing and contracts, CI/CD pipelines. It's very thorough. Let's pull a few of these just to get people want to learn more and grab in the book. So first one is, what do people need about contract testing as it applies to maybe web applications, since that's one of the chapters?

[00:16:00] Marie Cruz In chapter four, really we focus on the most common scenario where people will apply contract testing. This is when you have a web application. And it could be like a Rest API as a data provider. So in that chapter, we really showed step by step how you can write the contract test from both the consumer and the provider perspective. We've chosen Pact because it's the current de facto tool when it comes to contract testing. And it's something that Lewis and I have also used a lot in previous projects. If there are people who want to get started with Pact but they don't know how, so we provide the actual GitHub repo like step by step. We provide the different scenarios that are quite common between a web consumer and an API, and we walk them through how they would write the different configuration. Starting from defining where the Pact should be stored. And we've also done it in a way so that rather than setting up a Pact broker by themselves, because we want to make sure that they can immediately see the value of contract testing, we've chosen to use the free version of PactFlow, so they can just upload the contract there and see everything in action. Although, in later chapter we do have a chapter around the setup of how you can use your own like Pact broker if you're going to host it yourselves. But mainly that's the meat of chapter four, really showing the basic concepts. And then from the provider side, it's making sure that whenever they pull the contract that they can replay the interactions that the consumer has registered as part of that contract, and then making sure that the expected response coming in from the consumer matches their actual response. And that's where we sort of explain that if there are any miscommunication or if there are any mis like expectations, then that test can really capture those feedback earlier.

[00:18:24] Joe Colantonio Nice, so all the different types of contract testing you can do is one more difficult than the other or trickier like so comparing web as opposed to mobile or GraphQL, any one of those maybe require a little extra effort or it all depends.

[00:18:39] Lewis Prescott So yeah, I mean it's supported in all of them and they all follow the kind of same flow. So the implementation and the set up and everything should be quite similar. The only one that we kind of flag that requires that additional effort is event driven. If you are tightly coupled to the event system that you're using. So you would need to kind of refactor it or come up with way of making it agnostic to that technology. But that's the only thing if you're already tied into it. Otherwise you can just pick up and play with it as Pact test as all of the support, like languages or the main ones. And then, yeah, like GraphQL and stuff. It's got those plugins to enable you to do that.

[00:19:30] Marie Cruz Yeah, I think with the different tests, like what Lewis said, it all follows the same approach. However, in my opinion, I think the biggest area where teams start to find the whole concept of contract testing a bit more complicated is when they integrate it as part of their continuous integration pipeline. So that's why we have a whole chapter dedicated, and I think it's the longest chapter, if I'm not mistaken. We really covered step by step how to integrate the contract testing as a pipeline if you're using GitHub actions. We also, acknowledge that there are other continuous integration providers out there. We provided like snippets of say for example, if you're using CircleCI this might be an example config, but we focus with using GitHub actions as our main CI tool. And one other concept as well that me personally found a bit challenging in the past, so I wish someone would have showed me like the way to do it. I had to really do a lot of experimentation was if I need to trigger the provider verification automatically if there are changes in the contract. This is where having some sort of examples around that can really help the readers, because while the documentation out there online is all good, I think diving into it step by step will really help the readers like solidify why they need to also trigger the provider verification automatically. That's why we spent a lot of time in that particular chapter.

[00:21:19] Joe Colantonio And I guess before that one, you also have a chapter on storing the contract. Set an issue you see people struggling with where do they store them to make it more amount of secure scalable?

[00:21:29] Lewis Prescott Yeah, exactly. And it's one of those conversations that you have at the start, as well as whether you host your own contact store or you use the software as a service PactFlow offer. So yeah, we weigh up those pros and cons and also, yeah, we go through the kind of different functionality that you might need, depending on your circumstances. Whether that be authentication in a certain way, whether that be storing the contracts from in Docker or spinning off it in whatever format your company already uses.

[00:22:10] Joe Colantonio Yes. So do you see most people don't have contract testing tied into the CI/CD pipelines? Or do you see they try to and they fail. And that's why you created a chapter that's probably really one of the media ones in the book.

[00:22:23] Marie Cruz I mean, I can speak from personal experience, but the way I've experienced it was we would try to integrate like the basic sort of workflow. So if anyone looks at Pact documentation, they have this concept of Pact Nirvana. It's their way of sort of expressing the different levels that people need to achieve to make sure that they get the full benefit as possible with integrating contract test as part of pipeline. And when I first introduce contract test like I always go with the smallest value that I can provide which is okay, we can run one consumer tests. So we need to make sure that the provider is also verifying it. Great. That part is done. But at the same time, if there are any changes in the contract, what normally tend to happen was in the past, like the whole provider pipeline was being triggered manually from our side. So to reduce that manual sort of approach, we had to integrate this thing called Pact webhooks, which basically sends this message to the provider that, hey, there's some changes in the contract, but rather than triggering the entire provider pipeline, I'm only going to trigger the specific job to verify the changes that the contract has. So that has given us a lot of benefits. So there is no more manual intervention. But it took me quite some time to make sure that the webhooks are all configured correctly. There is a lot of like trial and error, so I just thought that other people might also experience the same issue because different CI workflows will have different ways of I guess in terms of their configuration. But the concept is all similar. I feel that that's quite an important piece of information to share, because other people might also experience the same issues where if there is a change from the consumer rather than triggering the entire pipeline, which I think depending on how your pipeline has been set up, that can take some time. So you would rather just have that immediate feedback loop of just triggering the specific provider verification job.

[00:24:46] Joe Colantonio Nice, so that brings us like like I said, people need to grab this books. There's so much info in this which jumping to part three now where you go over, provider driven contract testing. I think that's the last two chapters as well. Transitioning from integration test. What does that mean, Lewis? What's that all about?

[00:25:04] Lewis Prescott Yeah. So they're quite closely linked to last two chapters. So provider driven is kind of the approach of you've already got integration tests. You've already got your open API specifications and you want to retrofit in contract testing. And that's the scenario lots of people are in. Like they've already got the tests in place. They want to start improving what they've got. And so that's where provider driven comes in. You can use your open API specification. You can use the tests you already have, the integration test you already have. And then that translates them into a contract verification process. And so there's some adapters on the consumer side. So you can use the mocks you already have. And then they join up in the middle. And Pactflow has a magic contract comparison where it compares the mocks with your specifications. You don't need to have that duplicated effort of saying, okay, now I need to go write my contract tests. You can use what you already have. And in the last chapter where we talk about converting your integration tests, that's where we're talking about, okay, what benefits are you going to get from and what scenarios can be converted so that you can see what still needs to be integration test, because you still need some of those, what can be reduced in order to give you that earlier feedback and that stability of using contract testing.

[00:26:43] Joe Colantonio And Marie, let's wrap it up. Any thoughts on the last two chapters of what you were maybe excited about or something that you struggle with or people should know more about?

[00:26:51] Marie Cruz Yeah, I think in the past, I definitely struggled in introducing contract testing when we have like public API or third API or, third party APIs, where we really don't have much control in terms of their process because they have their own way of working, and especially with public APIs, where you don't really know who your consumers are. The consumer driven approach that we introduce might not be best suited for those scenarios. So that's why Lewis and I have also included a chapter or two around provider driven or bi directional contract testing, because we feel that now that's been introduced, there's really not much excuse for people to not look into contract testing because they can just take their existing test. From a provider side, it could be their existing open API spec, upload that to PactFlow. And then from the consumer side, whether you're using Cypress or I think they write this also supported? Lewis correct me if I'm wrong, but if you already have this thing functional tests, you can just use that as a basis and use the different plugins that PactFlow provide to convert that into a contract. So you can reuse the existing tests. And then what this bi directional concept will do is it will cross verify the two contracts together, one from the consumer and one from the provider, and check if the expectations are similar. I think this is a game changer because in the past there's been situations where while the whole concept of consumer driven really works, but in some scenarios it might not be the best approach. And this is where bi directional or provider driven contract testing can help. And I think, the people who are scared of contract testing then provider driven could be your answer.

[00:28:55] Joe Colantonio Love it. And like I said, this book is going to be the definitive guide to contract testing. You're going to want to get your hands on it. Like I still have a link to it in the comments down below for sure. It also has three appendixes where it goes over what tools you recommend, Project requirements specifications, this is going to have as a chapter on exercises, so it is a must grab. So really excited that you both wrote this. But Lewis and Marie, before we go, is it one piece of actionable advice you can give to someone to help them with their contract testing efforts? And what's the best way to find or contact you? Or get a hands on your new book on contract testing?

[00:29:29] Marie Cruz Yeah, I guess one piece of advice just grab the book and learn the different examples. But kidding aside, even if we're not talking about the book, just contract testing in general, it could be quite daunting to introduce a new testing type. But I think if you focus first in delivering the why. So why do you need contract testing? What is the value that you're going to get out of it? Then it's going to be much easier to convince the different stakeholders around the benefits and introducing these different tools. And also at the same time, I also want to just provide the advice that contract testing is not a replacement for actual communication between you and your teams. So contract testing can enhance it, but it can never replace the actual human communication. So you still need to communicate with your different data providers or different consumers. And yep, you can reach me via LinkedIn. I'm still on X but not very active. So LinkedIn would probably be the best medium. And I still have my blog testing with Marie.com, so I have my contact form there. If anyone. If you want to talk more about contract testing in general.

[00:30:47] Lewis Prescott Yeah, just echoing what Marie said, really like contract testing as a communication tool. And it's really great at that. And it really enables in-depth conversations around your communication between systems. So yeah, communication is key for me. And yeah, if you want to reach out again, reach out to me on LinkedIn or on X or find me on my website. Pactman.co.uk

[00:31:14] Joe Colantonio And as I mentioned, you definitely want to check out this book. And for our loyal TestGuild listeners, we have a special code for you use TestGuild24 to get a special. I think it's 40% off the book right now until August 24th, so definitely want to take advantage of it now. Check it out, let me know your thoughts and find it in the comments down below.

[00:31:33] Thanks again for your automation awesomeness. The links of everything we value we covered in this episode. Head in over to testguild.com/a500. And if the show has helped you in any way, why not rate it and review it in iTunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe, my mission is to help you succeed with creating end-to-end, full-stack automation awesomeness. As always, test everything and keep the good. Cheers.

[00:32:08] Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps, and automation professionals in the world, we'd love to have you join the FAM at Testguild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real-world value that can improve the skills or solve a problem for the Guild community. I love to hear from you head on over to testguild.info And let's make it happen.