Are AI Agents Replacing Contract Testing? DevOps Insights from Matt Fellows

[00:00:01] Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability for some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain Podcast and my goal is to help you create DevOps toolchain awesomeness.

[00:00:19] Hey, if you want to learn all about API contract testing and API observability, you're in the right place. We have the man in the house, Matt, who is the core maintainer of the popular API Contract Testing Framework PACT, and the co-founder and principal product manager of the enterprise Contract Testing Platform, PACTFLOW, that you probably all heard about, which was recently acquired by SmartBear. I guess I don't know about recently now. I think it was 2022. Prior to SmartBears, Matt really has consulted with all the top products or developed products of some of the top Australian largest brands, specializing distributed services, cloud computing, DevOps capabilities, and continuous delivery. He really knows his stuff, really excited about this episode. You don't want to miss it. Check it out.

[00:01:04] This episode is probably sponsored by the folks at SmartBear, empowering teams to build, test and deliver great software faster. Explore how SmartBear's bringing AI to the forefront of DevOps, quality engineering, helping organizations accelerate test creation, optimize automation, and to drive smarter decision making across your software development lifecycle. Check them out. Learn more using that special link in the comments down below.

[00:01:32] Joe Colantonio Hey Matt, welcome to The Guild.

[00:01:36] Matt Fellows Hey Joe, thanks for having me, really excited to be here. I've watched a lot of your episodes over the years, so it's great to be finally on it. Thank you.

[00:01:44] Joe Colantonio Yeah, you are the co-founder of PactFlow. How did you get into this? Like, did you plan on this? Like, how did you to get into testing? How did it get into contract testing and how did you get acquired by such an enterprise company, I guess?

[00:01:56] Matt Fellows Yeah, it's a great question. I don't know if I'd back you go, but like, I think like a lot of people, you kind of get into your own role in all kinds of strange ways back in school. I was a pretty sporty guy actually. I was going to do sports science or exercise in sports science. And you're kind of had a change of heart in the last moment and sort of pivoted towards computer science. But I did play a lot video games in high school. I did a bit of basic coding. I remember, a friend of mine gave me a Debian disk, the physical disk of Debian and I was really excited to install it and I got home and put it in there and I'm like, what is this terminal window thing? Like I was so used to using windows at that point. And so, yeah, I kind of got interested in computers at that point, so I was kind of a weird growing up, like very sporty, but also kind of, a bit nerdy in that sense. And then, last met like many people. You didn't really know what you wanted to do at that age. I'm sure it's that's still common. I ended up doing a double degree in computer science and cognitive science, so a bit of a strange one. Back then, this is 20 odd years ago, that was sort of trying to put together, effectively artificial intelligence. It's like take the human brain, take the computer capabilities and try to make something intelligent out of it. Of course, things have moved on a lot since then. But then straight out of university, I got a job. Doing web development and did that for quite a long time, about 10 years, ran a team there, and I really just wanted to get back into, you eventually get far away enough, you become a manager, and I wanted to really get my hands back in the weeds, so I did some consulting. And that's where my first consulting job, actually, I came across PACT. It was a pretty new thing, and I'd use it on this client, and it was a very transformative project, actually, it was out at Australia's largest job board over here called Seek, a fantastic company, and we sort of did this called DevOps transformation, went from monolith to microservices. This is circa 2014. And we took their first product and took it to the public cloud onto Amazon. And we went from deploying once every couple of months, really painfully, to literally, we could hit a button, push the code, seven minutes later, it was in front of a customer. And I still remember like in Melbourne, with the sort of head of the Australian division was walking around and we did a bit of a walkthrough of what we did. And we'd like to build lights, the old school things. And we got him to actually sit down and make a change to the homepage of the application. And he just had no idea what was going on. Like it's seven minutes, it's like, I don't understand. Is this like a test environment was like, no, that's production, you're in production. And so that kind of really helped cement fast continuous delivery type activities in that company. And it was very formative for me in my consulting journey. That's when I got involved with PACT. I thought it was a great tool, but of course, like many open source things. It needed a lot of work. It was pretty rough around the edges. I started maintaining it at that point. They're about 10 years ago and have been involved with PACT ever since. In terms of the question, how do we build a company and how did that happen? I liken it a little bit to boiling a frog. It sort of gradually happened. Again, when I was at the consulting company, we do a lot product development, like the very product focus, lean startup, that kind of thing. Like let's iterate, get things in front of users, get feedback and go from there. And I guess you work for enough customers and companies and you get frustrated at some of their decisions. You're like, I want to have to make that decision. I want skin in the game. And we worked with the directors of DiUS was a company and they're a great bunch of guys and a really, really good company, very supportive of this. We kind of gradually, introduced it just a free plan where you could sort of host these contracts. And yeah, it's kind of the rest is history. A lot of people started using it. We got a bit more investment from DiUS to help us to build it and yeah in 2022, we got a knock on the door from SmartBear and as we say, yeah, the rest is history.

[00:05:46] Joe Colantonio Love it, love it. A lot of pivots there from sports science, artificial intelligence to contract testing. How'd you know artificial intelligence was going to be a thing or did you not know you just was interested in?

[00:05:58] Matt Fellows Yeah, it's a really good point. I didn't actually think too much about the artificial intelligence side of things when I joined the course. I actually really enjoyed the psychology side of things and human-computer interactions and all this. It just happened to be that the course had a lot of philosophy and morality centered in the middle. But I think also just I didn’t know what to pick at the time. I found both computer science really interesting and I really like the cognitive science of things. It had neuroscience in there as well. It was just a really interesting course. At the time, I had to build our own neural nets and these things from scratch and perceptron networks, etc. And so I found that, again, very technically, scientifically, theoretically fascinating. In terms of real-world applications at that point in time, definitely we weren't thinking of artificial general intelligence, Generative AI was definitely not a thing. It sort of had its roots in sort of what deep learning would be these days in terms of those really deep networks. It was more about classification and identification of things and solving those very specific point problems, definitely wasn't about what we're seeing today. And obviously, we've always had a view to the AGI world at some point, like it's always been somewhat this futuristic thing. So yeah, in my particular case, I just really enjoyed it from a theoretical interesting point of view, but then it kind of went away, like you get a job and all of a sudden the practicalities of life, all the practicality of working in a software company, it kicks in. And you have to deal with tests and all these other things and all of those theoretical things go away.

[00:07:29] Joe Colantonio Absolutely. I'm sure it gives you insight, though, into generative AI and where things are going because you have a solid background, it sounds like, for sure.

[00:07:37] Matt Fellows Yeah, I mean, we spent a lot of time doing like fuzzy logic and building our own perceptual networks and actually remember, I was thinking of the other day when I was doing something around the Agentic AI is that I remember doing this thing called agent oriented programming. And I was think I should probably go back and look at that. I wonder if there's anything in there that would be interesting or that would relevant for today's world with what is now what we refer to as Agentic AI. I should go back. I've got some textbooks somewhere that probably have something about it. But yeah, in terms of that background, I think, you learn about obviously Turing, you spend a lot of time thinking about Turing and you think about the morality and all of this. So I think a lot of that grounding has been helpful with some of my thinking today. But of course, the world moves very quickly. And particularly in the last few years, GenAI has moved very, very quickly.

[00:08:24] Joe Colantonio For sure. I didn't mention your bio, but it is part of your bio. You describe yourself as someone that I think really enjoys the intersection of tech, humans, and ideas, maybe, how does that mindset help shape your approach to product and open source and how you see the future with AI being more and more involved in everyone's lives?

[00:08:42] Matt Fellows Yeah, no, it's a good question. I think like a lot of people who've been around for a while, you sort of swing a little bit between being a practitioner and getting really deep in the weeds and then, sometimes you spend so much time that you become disconnected from the thing you're trying to do, the customer or the problem or what have you, and then you move up the chain a little bit and then feel like you're too far away from the code again. There's this manager, engineer pendulum, a few really good articles out there from charity majors on that topic. That really resonated with me. I think on the one hand, I really like the problem solving aspects of things. I love the coding aspects of all that as well. But from a product point of view, you do need to make sure that you're connected to the humans at the other end of it who are going to use it. Maybe not so much these days, but at least traditionally someone's going to be consuming your product. And so I think being at that intersection is really interesting. And ultimately, again, going back to that sort of formative experience I talked about earlier. As software engineers or as developers, testers, what have you, you want to make sure the thing you build gets in front of people and gets out to the real world, because what, yes, there's something interesting sometimes in doing something just purely technical, but if no one's using it, is that fulfilling? Not really. I think being able to connect real world problems, real people to the things you're building is really important. That's probably the first part of it. I think the second part you're asking about sort of how AI is coming into that, was that the question, Joe?

[00:10:10] Joe Colantonio Yeah, definitely.

[00:10:11] Matt Fellows Yeah, I mean, it is interesting, obviously, like, there was a survey from the Stack Overflow survey came out yesterday. I'm not sure if you've read that yet, but yeah, it just came out yesterday, so I look at that for a few reasons. It's a really good, really good insight into the minds of those who are consuming our types of products as a developer tooling company, who's using it, why they're using it. What other tools they're using. There's a whole section on AI, unsurprisingly, because it's such a big topic at the moment. I think it's okay to summarize it like this that Sentiment has actually gone down in terms of trust and frustrations with AI, getting it close, but not close enough. It's kind of right, but you have to spend more time debugging it. And I think what they say is compared to last year, those numbers have gone down a little bit. That being said, the usage is continuing to go up and everyone basically says we expect to do more of it next year. I think that's really interesting. It tells us maybe the hype is, Hype is the wrong word, but it is very hyped up. It is very much in everyone's face. If you look at those, one other statistic that I found really interesting was when they evaluated tools, which tools to use, they asked a whole bunch of like, what is the most important to least important set of priorities for you when you're using these tools? And out of 10, AI capable was number 9 on the list. I think number one was ease of use, I think it was. And two was basically the capabilities. I think as tool builders, as really as anyone. But if you're building tools, that tells you some things like AI is interesting. But we still fundamentally care most about is it easy to adopt and use. And does it have the features we need to do the job. I found that a really interesting data point. And obviously, we'll have to unpack the entire survey. But that was probably one of the headlines for me. Coming back to how it changes everything, I think it's obviously going to change things like the Agentic movement into you're working with a pairing partner that you can give jobs and it can often go off and do in the background for you or you're pairing with it in real time. We're gonna see more of that, obviously. But ultimately, maybe I'm a little bit too optimistic, but the human in the loop design. I think we still want to retain that. Humans are still the ones making those decisions. We're still the one's getting paged in the middle of the night when something goes wrong. Do you want a machine creating code that's gonna then wake you up in the night? I don't think so. All those factors, I think still put the human at the center and the arbiter of that quality and so yeah I think more augmentation is going to be part of the future. I don't think it's going to solve the biggest problems though just yet to just to go off on one more tangent. I'm not sure if you're familiar with Goldratt's work on the book called The Goal but it's a theory of constraints is what we're getting at here and if you look at productivity at the individual level, AI is helping us all do things more quickly. I think that's really clear and that was also that came out in the survey. But at the company level, a lot like that hasn't seemed to have filtered through just yet in terms of companies being much more productive. And I think there's a bunch of reasons for that one is that if you're just optimizing at the individual level, you haven't necessarily solved the systemic problems. And often the constraint isn't the individual performance. It's actually the system itself. I think we're going to have to address that before it really makes a big difference. And the second factor I think is that, a lot of companies are still actually experimenting and trying to work out how to get the most out of these types of tools. There's some good public examples like Shopify, who have gone pretty hard at this and there's a bunch of others as well. But that's going to take some time, I think, to filter through as well.

[00:13:43] Joe Colantonio Absolutely. All right, so let's level step before we go a little bit further. We mentioned contract testing a few times, but the folks that don't know what is contract testing and has it pivoted in the age of AI or is it still the same? Do you see AI impacting it where it's not a thing anymore? I don't if that makes sense.

[00:14:01] Matt Fellows Yeah, it's a really good question. I guess coming back to basics, what is contract testing? So by analogy, my colleague came up with this years ago, which is, do you test that your fire alarm works by setting your house on fire? Or do you press the button on the fire alarm to check that it's working? And of course, most people would answer, I don't set my house on a fire. Contract testing really is a way to ensure that two different systems, such as microservices or a mobile app and an API, are compatible with one another. And it's going to capture the interactions that occur between the two of those systems, and then we can into a contract, and then, we can use that artifact to check that both those systems adhere to it over time. And that's very abstract. PACT is one version of that that allows you to do this kind of testing. It's specifically a code-first way. You write a unit test against your API client. That spits out, at the end of it, all the interactions you made with it, and some other things like matching rules, et cetera, to help make it more flexible. And then that contract is what gets shared between its provider, who then verifies it. So it's a bit like record and replay in that sense, but it's got matching rules and these other examples attached to it to help make the interactions between them very concrete. We would also refer to that as a specification by example. You could do that without the need for a specification. You've got all the examples that sort of explain how the thing works. It's often compared to end-to-end testing. And we're responsible for a lot of that. End-to-end testing is where you stand up the entire system. And you run tests from the top to bottom. And I'm sure your audience is more and very familiar with all that. Contract testing differentiates itself a little bit from that in the sense that you want to isolate the two components that are talking to each other and take it out of the broader system. And so that does enable you to do that in terms of the way you scale that up across the different systems. With an end-to-end system, if you're running the test from the outside, the number of tests you need to cover it becomes quite significantly large. And so you have to be quite tactical in terms what end points you test. Whereas with contract testing, it scales linearly, so you can keep adding as many contract testers you want between those individual services. And you're just, it's an N plus one as opposed to constantly having to combinatorially work out all the interactions. That's a bit complicated. Let me come back to basics. Your question is, where does contract testing go in the age of AI? Well, yeah, that's the million dollar question. Of course, the product managers like myself are thinking. I guess there's a few parts to that. The first answer to that question is, well, we know AI adoption is going to continually grow. That's not going to stop anytime soon. We went out from the survey, but I think we all just know from our own daily use of it that there's more and more tools coming up that are doing this. I think that's the first point. I guess the second point is we know that the LLMs are going to get more and powerful, the frontier ones in particular, and probably more frontier LLM will start to appear soon too. That have very specific use cases. I saw one the other day specifically for note taking, and I think we'll start to see more of that as well. I do think running PACT tests might be a thing of the past, eventually for example with PACT, but more generally. I think the future is moving towards autonomous contract testing with that human in the loop design, where effectively those things can be written from your existing code or other artifacts. What I will say is, and then we'll probably, everyone's going to be grappling with to some extent, as a community, one of the things, we know the tests are often they're used there for regression. They're used to make sure behavior doesn't break, things keep working. That's an obvious use case for tests. But the thing that I think we often miss in this conversation is that tests aren't there just for that one purpose, particularly unit tests. Unit tests are there. And in the case of PACT, the reason I mentioned this is PACT is often written as a unit test or should be written as unit test, or close enough to. By automating unit tests, that's an interesting thing for me as someone a practitioner of XP and these kinds of things because unit tests exist to not just check behavior, but to help you refactor your code, to help your build higher quality code. It's testing the system design itself. And so if we auto-generate all those tests, are we losing something by not doing that? Like are we making the systems less good? Are we making them worse, less quality?

[00:18:25] Joe Colantonio Are we if we do that?

[00:18:28] Matt Fellows I don't know. I think the jury is still out on that a little bit, but it is absolutely certain that we're vibe coding all these things. We're not necessarily producing better code with AI, right? I think there's definitely a risk of producing that. And again, the Stack Overflow survey has some little insights into the differences between those who are learning to code versus those who have been professionally using it. There's a few data points now. I won't remember off the top of my head, but what i worry about of course is it the more more things we throw at the wall through vibe coding and don't check ourselves, we're going to be creating more and more bugs that we did before. I've gone blank on the old saying, but for every line of code I write two bugs, I'll have a job forever. Well, I don't know what the equivalent of that is for vibe coding, but I suspect it's more than what you're doing if you're writing by hand. I do worry about us producing worse code and by definition, if we're doing the same thing to produce tests, are we now making our test worse as well?

[00:19:24] Joe Colantonio I have the same problem. I've been vibe coding for a few months now, and it's just pumping it out to production. And just today, it just AI wrote something crazy and it brought the whole site down. And this is just my little application. I can imagine how many people actually doing this. I haven't even looked at the code and I'm not a tester. It can be kind of dangerous just putting your full trust in it almost without having that sanity. And every time I bring this up to an AI expert, they go, oh, just early eight days now though. A year or so, two years, it'll be even better where you don't have to worry about it. But I don't know if we're ever going to get there.

[00:19:57] Matt Fellows Yeah, no, it's absolutely like, I think it will get better. Like I would agree with that. But if we expect AI to get to human level intelligence, well, we know humans make lots of bugs, so then we expect at least the clones of us to do something similar, maybe they'll make the point that AI will exceed human performance for some of these things. And maybe that's true as well. Will they be bug free? I don't know. I don't think so. And then also that bugs only one dimension as well, right? There's performance and all these kinds of things. And of course you can go, we'll go get Claude Flow or something like that out, and we'll put a swarm of agents on this problem and make it performant. And so maybe that will be part of the future, but I do think who checks a checker type thing. I don't know if we're gonna live in a world without that. And again, going back to all the other practical limitations, at least for now, if the banking system goes down, mostly humans still care about it. And so it would want to come back up pretty quickly. Whereas if we start optimizing for machines, certain end goals might fall by the wayside. And they might fix it, but they might fixed it really slowly because they want to re-optimize the system. I think it's going to be hard to pull the humans out of the loop, at least for some while. And as soon as humans are there as part of an important part of that value chain, I worry about putting a full confidence in AI at this point.

[00:21:16] Joe Colantonio I definitely agree. All right, so as you mentioned, we see AI generating more and more code. Could AI generate reliable contracts and mock automatically, which it sounds like maybe? And does that introduce new risk, which I think we touched on a little bit more.

[00:21:31] Matt Fellows Yeah. So specifically on that question, we in Pactflow will definitely be moving more in that direction for sure. I think our customers tell us pretty clearly, and it's no secret, we know the pros and cons of tools like Pact and the challenges that come with writing Pact tests. We get told by our customers and competitors that writing PACT tests is a maintenance burden. Before I address the AI question, what I will say is I was talking to a customer a few weeks back. And we're exploring some different options in terms of how we're looking to look at like a PACT 2.0 in the future. And this person made the point that they really liked the way the PACT works today in that you have to write a unit test and you have slow down to write it, which was interesting. Like it's not something that's totally surprising, but it's interesting to get that kind of contrarian view. And their point was like, you have think through what you're doing to make sure that your code is doing what it's supposed to do. I don't necessarily want to be in a world where those tests are generated automatically for the reasons we kind of discussed. In that, how do I know that it's really stressing the contract stressing the system? It's high quality. Our code is actually doing what it should do. Like, is our API client supposed to do that? Maybe it's not. And so, of course you run the risk then of automating tests for a part of your system that doesn't get used. And so now you've just got these wasted tests. I think that's one angle. But in terms of where does AI go here? Yes, I do think we'd be moving more to that sort of autonomous testing with the human in the loop design. So you'd imagine in the case of PACT, for example, we would produce those consumer contracts that fit to the API client and we would make sure that they would align to best practices, et cetera. And then the contract would be able to be generated from that. And conversely on the provider side. We would set up the provider side, those tests as well, pull down all the things you need to test and point that against the running version of the provider. Some of those things are tricky and complicated and that's where tools like CoPilot and Claude, et cetera. Okay, so pick whichever one of your agentic tools that you like, can do a really good job with that because they have really good awareness of your code base. They can index things quickly, they can do that. That's something that those tools are really good at. I think products like us, developer tools like us will spend more and more time investing in things like MCP, which is something we're doing right now, to not only provide capabilities to do certain things like generate a test or review a test, or provide access to existing APIs so that you can introspect data or introspect contracts and things to help write them and verify them, but also to provide specific prompts that enable those co-pilots and things, to do specific use cases. For example, establish PACT on a new project. That's a very complicated thing to try to automate if you're not a frontier LLM. We don't have that kind of capability, you would need to call into those really powerful models. Whereas the agents like copilot, Claude, Cursor, et cetera, they can do this. And so if we can give them the right prompts with the right information and context, they're gonna do a much better job of automating some of those. So that's probably where headed. At least from the existing point of view. And of course, the challenge we want to make sure is that we don't have tests that just check, just sort of fit what they are currently doing. And so effectively, you're just testing the bugs as opposed to finding the bugs. And that's probably part of the challenge. Look, it's entirely possible that in some future world, you don't need something like PACT or any of our alternatives because you can just statically look at code on both sides or something and go, actually, we know what this API client can do on this side. We know what the API can do, and we can just tell you, yep, they're going to work. And maybe that's what we eventually get.

[00:25:13] Joe Colantonio Yeah, when you bring up MPC, like a Playwright, you don't need to know what the clicks and everything is. It just knows. You don't even know. Just say, write me an automated test and may call that that server and know what to do. It's almost like contract testing and I'll know like I just know what they're expecting and what I should send. It sounds like.

[00:25:31] Matt Fellows Yeah, definitely. I think the Playwright example is a really good one even the Cypress, I think that those analogies work pretty well for us because you can't really write those types of tests without access to the code base. Like you can do it from, well, you kind of can do it from a UI, I suppose, but you need to run it in some real environment. It can't just be done through a GUI somewhere else. Doesn't have access to your code. And so same with PACT testing or contract testing, we definitely want to make sure that we check the real behavior of your API client and not just check the API itself. We want to make sure, okay, if you've got a mobile application and it's calling these different APIs, that we actually check the mobile client calls those various APIs to make sure that we've got visibility. And this is something I didn't mention earlier, but I think one of the benefits of contract testing over just API-focused testing, I would say schema-based contract testing where someone's just checking the API matches its specification. One of the benefit of contract testing is you get visibility into what your consumers are using of that API, what's the surface area that's being used. In the case of PACT, you can get that down to the field level. And that's really powerful because now we can know, okay, we've got these 10 consumers, but no one's using this endpoint or no one is using these properties. We can refactor those out later on without damage. That's a really powerful thing to have. A bit of a sidebar there, but coming back to the contract testing, AI sort of things, we obviously wanna make sure we're testing the clients as much as we can and ensuring that we're tested real behavior. And then, of course, once we've got the test for real behavior, then we've go the confidence. I would imagine the kinds of sources I can think of that we would be able to use to generate those, because all of those tests happen API to API. Some of our competitors, for example, they don't do contract testing, but they do testing through, effectively, kind of an end-to-end test by using traffic analysis. Hyper-test and signal data are examples of companies to do that kind of other ephemeral environments or mocking using traffic inspection and effectively record and replay, but at scale. eBPF is another promising kind of technology. If you guys come across that and it's a bit esoteric at the moment. It doesn't quite work fully across platform, but I think a lot of really cool things will happen in that space in the future. And obviously a whole bunch of things have already happened. Hotel open telemetry might be another source to pull that kind of information from. Hotel by default doesn't capture full requests, response payloads, et cetera. If you want to get the full details, you'd probably need to modify your instrumentation. But there are a few kind of alternative promising ways that probably could be pulled into, we'll call it, quote unquote, contract testing workflows, whether that's with PACT or otherwise, to check that both sides are working together.

[00:28:13] Joe Colantonio This might be a dumb question. Do these Agentic, not these Agentics, these code creating things like Cursor and Copilot, do they know about contract testing? What does the person like, it knows like I'm gonna create a contract test for this or you have to prompt it or actually have it in the instructions every time you write a code, make sure to write a contract test for it.

[00:28:35] Matt Fellows Yeah, for the most part they do. They're usually built on top of the frontier models, so the frontier models, like OpenAI and Claude, et cetera, and more anthropic and all their models. They obviously index the internet at a certain point in time. One of the challenges is they often have outdated, DSLs and things like this. I think it was OpenAI, but one of them was constantly using a very old DSL and we couldn't force it to use a newer one. And so that's where additional work on our side to get it to generate tests is required to do that. And also sometimes, because the APIs and the DSLs that exist out there, at least in PACT world, there's a lot of them. And there's like a generation of changes. It doesn't necessarily know which one is the right one to use. And it often will use the one that has the most examples on the internet. Something I noticed, for example, is in the JavaScript land, a lot of the examples I wrote because I managed PACTJS. And in my test, I still use Mocha, and I still used Chai as my testing framework in the assertion libraries. And it's really interesting because if you look at the kinds of tests that when you generate with OpenAI, for example, it always, with JavaScript, defaults to using Mocha and defaults to using Chai, as the assertive language, which would tell you it.

[00:29:56] Joe Colantonio Probably trained on you and your work.

[00:29:58] Matt Fellows It's trained on those examples exactly. And they're probably in some ways not the best because most people probably use Jest these days. And so having it default to Jest is again, we've had to put work in there to make sure that we don't do that. And so I think that's a good reminder that they are built on our own work. And so we need to think about kind of, I was talking the other day to our dev team, it's like, do we need think about effectively SEO, but for AI now, like if you think about in the old days, you needed to get a lot of content out there so that the search engines would find you. It's kind of the equivalent of that now with AI. We want to make sure we've got really good examples and documentation and LLM.txt, et cetera, out there. So the agents index your code and your examples in a way that you want them to, and to make them visible, to then have them used in the tools. It's going to be an interesting world there. But yeah, to come back to your question specifically, they do know about contract testing. They occasionally just need, well, they usually need a bit of prompting and steering to get the right kind of output. And so in some of the tools we've built, there's a fair amount of prompt engineering we've got in there to steer it away from bad practices, to steer towards good practices, to review the code in a way that we would like it to as a humans. We just released code review this week, actually, to help anyone who's got existing tests to make sure they're good, but also when we generate tests to make sure that the generation itself follows those best practices. There's a lot of little things in there just to improve the usability of that, but for the most part they can do a pretty good job if you prompt it really well.

[00:31:28] Joe Colantonio Can you share anything on maybe on the roadmap for SmartBear, what you see contract testing? I know you could not see in the future, but three to five years. I know, you mentioned the stack overflow where they said people like AI was 10th on the list. So how do you take that with where your roadmap is, which I assume is just AI heavy to begin with.

[00:31:47] Matt Fellows Yeah, it's a really good question. I've been thinking about this a lot recently, as I'm sure pretty much every product manager has for a while now as well. So I think the first thing to point out is, yes, we need to enable our tools in the AI ecosystem. So that would be what we've done initially is built some very base capabilities in the space for generating PACT tests, reviewing PACT tests, and across multiple languages, et cetera. We're never going to be able to do that and compete with the frontier models, right? All we can really do is work with them because unless you've got thousands of engineers with PhDs and giant data centers, that's a very difficult thing to do. I think the best thing for us is to PACT with those types of models and leverage the best we can out of them. What we need to move to, where we are moving to now is supporting work with an agentic system. How do we make our data? How do we get unique sources of data into our products? Because that's actually what's gonna be really helpful to provide support for those agents. When you're running tests and you're fixing bugs, hey, we've got a bug snag integration. We had this error in production. Can you help me fix this error? In your agent, you give it the error. It goes to bug snags, finds the error, finds a line of code that is relaxing, and then it tells you, okay, you need to fix this line of codes. And so those kinds of cross product journeys within SmartBear, for example, they're going to be really powerful in terms of solving really big issues, especially if you've got contract tests there and you've got your spec and you got your other tests all in one place. Now the agent can query all that data from those various systems within SmartBear, piece together the puzzle, and then suggest the next step to help either fix a bug or modify that change to the API. If you go a little bit further into the future, obviously three to five years is like a millennia in terms of AI. But I think we all need to be at the point where we've got really good support for those agentic workflows. That's the first thing. But beyond that, again, going back to the Stack Overflow survey, what everyone tells us, more breadth and depth of feature support is number two, and ease of use is number one. And so I do think getting back to bread and butter stuff is actually going to be where we win in the future. So we'll need to better support the models. Would need to be in that ecosystem through MCP or some of the other protocols that exist in there. But ultimately, it's actually building really good products that have the features that solve the problems. And then humans and agents, hopefully, will then choose those features that have those products that have the most capability.

[00:34:26] Joe Colantonio The first part, you did mention writing for agents instead of humans almost. It's the same thing with SEO now. I heard you have to write for, be able to have the LLMs be able to understand it rather than a human. It's a hard mix, I would assume, being a product owner and product manager of that, for sure.

[00:34:40] Matt Fellows Yeah, I certainly worry about that in some ways, but it makes sense, at least in the near term, it makes sense to ensure that you're giving your product and your capabilities the best chance of being used in those tools. I still think humans are first, but I still use please and thank you when I talk to the agents, just in case, you never know.

[00:34:59] Matt Fellows I heard that waste tokens and waste energy when you do that. Awesome. Okay, Matt, before we go, is it one piece of actionable advice you can give to someone to help them with their API contract testing DevOps efforts? And what's the best way to find or contact you or learn more about PACT and contract testing and all the goodness at SmartBear?

[00:35:16] Matt Fellows Yeah, I mean, the simplest advice for me is just start simple and get started. So if you want to look at contract testing, don't overthink it. Just literally get in there, write your first test, make a few mistakes and then move on and keep it small and constrained. I always think that's the best way to get started, I'm a very kinesthetic kind of learner. I like to just get my hands on things and figure it out that way. That's how I like do it. In terms of getting in contact with me, I think I've got all the usual things I want linked in. You better find me under my very interesting name of I think it's digital Matt actually and then you can find me under a less interesting name which is Matthew Fellows.

[00:35:52] Awesome. We'll have a link for that in the comments down below. And for links of everything of value, we covered in this DevOps toolchain show. Head on over to TestGuild.com/p198. That's it for this episode of the DevOps Toolchain show, I'm Joe. My mission is to help you succeed in creating end-to-end full-stack DevOps toolchain awesomeness. As always, test everything and keep the good. Cheers

[00:36:18] Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation, testing, and DevOps. If you haven't yet, join our vibrant community at TestGuild.com where you become part of our elite circle driving innovation, software testing, and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges, we're excited to collaborate. Visit TestGuild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve.

[00:37:02] Oh, the Test Guild Automation Testing podcast. With lutes and lyres, the bards began their song. A tune of knowledge, a melody of code. Through the air it spread, like wildfire through the land. Guiding testers, showing them the secrets to behold.