About This Episode:
Flaky Playwright tests got you down? Discover Vibe Testing, a new AI-driven approach that lets Playwright tests understand design intent, adapt to UI changes, and self-heal intelligently.
In this episode, Joe Colantonio talks with Vasusen Patil, Co-Founder and CEO of Donobu, about how their platform extends Playwright with AI-powered “Vibe Testing.” You’ll discover how this approach blends visual assertions with contextual understanding to build resilient, low-flake tests that keep shipping smooth.
You’ll take away:
What “Vibe Testing” really means and why it’s a game-changer
How AI-authored Playwright tests can self-heal without false positives
The key to balancing autonomy with tester control
Why Donobu’s local-first model keeps your data safe while cutting test flakiness under 2 %
How to try Donobu’s free Playwright AI toolkit
If you want to see where test automation is heading next — and how to future-proof your QA career — don’t miss this one.
Exclusive Sponsor
Discover TestGuild – a vibrant community of over 40k of the world's most innovative and dedicated Automation testers. This dynamic collective is at the forefront of the industry, curating and sharing the most effective tools, cutting-edge software, profound knowledge, and unparalleled services specifically for test automation.
We believe in collaboration and value the power of collective knowledge. If you're as passionate about automation testing as we are and have a solution, tool, or service that can enhance the skills of our members or address a critical problem, we want to hear from you.
Take the first step towards transforming your and our community's future. Check out our done-for-you services awareness and lead generation demand packages, and let's explore the awesome possibilities together now https://testguild.com/mediakit
About Vasusen Patil
Vasusen Patil is the Co-founder and CEO of Donobu — an AI QA platform that extends Playwright. Previously, he led Consumer Engineering at Coursera, where he grew revenues 10x to nearly $700 million and helped guide the company through its IPO.
Connect with Vasusen Patil
-  
- Company: www.donobu.com
 - Blog: www.donobu.com/blog
 - LinkedIn: vasusenpatil
 - Twitter: @vasusen
 - YouTube: www.youtube.com/channel/UCSx4EYBiYZ52_BV9dcnce-g
 
 
Rate and Review TestGuild
Thanks again for listening to the show. If it has helped you in any way, shape, or form, please share it using the social media buttons you see on the page. Additionally, reviews for the podcast on iTunes are extremely helpful and greatly appreciated! They do matter in the rankings of the show and I read each and every one of them.
[00:00:02] In a land of testers, far and wide they journeyed. Seeking answers, seeking skills, seeking a better way. Through the hills they wandered, through treacherous terrain. But then they heard a tale, a podcast they had to obey. Oh, the Test Guild Automation Testing podcast. Guiding testers with automation awesomeness. From ancient realms to modern days, they lead the way. Oh, the Test Guild Automation Testing podcast. With lutes and lyres, the bards began their song. A tune of knowledge, a melody of code. Through the air it spread, like wildfire through the land. Guiding testers, showing them the secrets to behold. Oh, the Test Guild Automation Testing podcast. Guiding testers with automation awesomeness. From ancient realms to modern days, they lead the way. Oh, the Test Guild Automation Testing podcast. Oh, the Test Guild Automation Testing podcast. With lutes and lyres, the bards began their song. A tune of knowledge, a melody of code. Through the air it spread, like wildfire through the land. Guiding testers, showing them the secrets to behold.
[00:00:35] Hey, a flaky Playwright test wasting cycles. Today, we're going to show you how to make them self-healing with AI, what actually works in production, what breaks and how to measure the wind. Today, we're talking all about it with the founder of Donobu, an AI QA platform that extends Playwright. Really excited about this. We actually met at Test Guild IRL in Chicago and really got into AI and how it can help testers, you don't want to miss this episode, check it out.
[00:01:00] Joe Colantonio Hey, welcome to The Guild.
[00:01:03] Vasusen Patil Hey, thanks, Joe, really exciting to be here and it was really great to meet you and others at our first ever Test Guild conference in Chicago.
[00:01:13] Joe Colantonio Absolutely. So what I thought was interesting right off the bat is I asked a question, how many people actually using AI or Chat GPT or anything, any type of LLM with testing and only four people raised their hands, two of them, one was you, one was Jason Huggins and the other two I didn't know. So I was kind of shocked by that. What were your thoughts? I'm just curious to know because I didn't ask you what your thoughts were on that. I thought that was kind of weird.
[00:01:39] Vasusen Patil Yeah. To be honest, I was pretty stunned by that. I looked around the room. I had assumed that about half of the people in that room would raise their hands at least because you even asked a simpler question. How many of you have even at least vibe code something? And I was so surprised by that because it's so easy and free to vibe code now that it's surprising. And I realized how much of a bubble I live in Silicon Valley.
[00:02:06] Joe Colantonio So yeah, just curious to know then, I want to get into why you found that shocking and maybe what you did to maybe help people get there, but first, maybe a little background, how you got into automation?
[00:02:18] Vasusen Patil Yeah, for sure. I've always been an engineer. I moved to Silicon Valley to join a Y Combinator startup. And I wrote my first ever automations there a long time ago. And this is when it was like RSpec and Selenium and you just went at it. After that, I led engineering teams at multiple places and I was leading consumer engineering at Coursera. There, I never wrote automations, but my team had to write a ton of automations to keep the Coursera platform consistent globally.
[00:02:57] Joe Colantonio Nice. How then did you get into Playwright? There were so many frameworks to choose from, but we're going to get into maybe the solution you built on top of it. But first, why Playwright?
[00:03:06] Vasusen Patil This is a good sidebar that maybe I should mention. And I told this to Jason. My biggest project, what got me my first job, I would say, was I had scraped all of Wikipedia and all of New York Times on my college MacBook with Selenium in 2010. And this was before AI. So I was always very excited about it, but as you can imagine, it needed a lot of wait statements and all sorts of drama to stop all these browsers from detecting you as a bot. And frankly, it took like months. But it was worth it. Everybody loved the project. I got tons of great data because of it. So I've always kept in touch with all the frameworks. At Coursera, we had moved to Puppeteer when Puppeteer came out. And we still had quite a bit of problems, especially because we wanted to test Edge and other devices as we were becoming more enterprise and much of the world uses Microsoft. When Playwright came along, it was one of the few things I always keep in touch with automation. It was another of the first things I tried. And I've been pretty hooked. I also think I know some of the team members who moved from, I think, Puppeteer to Playwright here. So there was just a nice confluence of things.
[00:04:28] Joe Colantonio Cool. Why then, we're building up on your career. What made you start think, oh yeah, I'm a startup. A startup, not only is it gonna be a startup, but it's gonna be startup around automation and Playwright.
[00:04:41] Vasusen Patil That's a good question. I think I've always wanted to solve real problems. And that's what I've done at all of the companies I've worked at. At Coursera's engineering, I would say every month we had to tackle a new and hard productivity problem and everything went crazier once AI came on the scene. I was tasked by the CEO to basically be the AI czar internally, just get knowledge flowing, get every team excited, get them using stuff hands on. And I was doing the same, right? So I would say what Tricker did was just seeing how terrible the testing landscape was going to become very soon. Because, it doesn't take that long to realize, holy shit, everyone's going to be writing a lot of code, many of them are vibe coding. And the worst is-and this I really truly believe it hasn't quite panned out, but most software is going from deterministic software to partially autonomous software. And that is key currently in the world. If you ask any good testing team, they have no idea how to test partially autonomous software. My most embarrassing moment came when my team was testing Coursera Coach, which is our chatbot. And I remember I went into one of these dog fooding meetings. And I tested it. I was assured everything was great. And all I said was like, Hey, what other courses do you recommend coach? And coach said, these are the three links for Udemy, which is our direct competitor. And I was asking the test team, like, how are we testing this? And their whole plan was to have an actual word they were testing in there for the competitors name going forward so that it never responds with. You realize that all of this is going to get crazier once AI comes in the scene and most software starts becoming partially autonomous.
[00:06:45] Joe Colantonio How do you test it though? That's the that's the thing. And a lot of testers are fighting this. Like you can't test something that's not deterministic, but actually that's what we're heading towards. How do we even start that or tackle that mindset?
[00:06:59] Vasusen Patil Correct. So I have a strong belief that when software becomes partially autonomous, your test suite has to become partially autonomous as well. You cannot test partially autonomous software with deterministic tests. That's the key. So then you fight fire with fire, you add AI on this side, and you start testing.
[00:07:23] Joe Colantonio Love it. How did you convince investors to invest in your company and what was the pitch?
[00:07:27] Vasusen Patil It was actually pretty simple like hey, I have been shipping software now for almost 14-15 years professionally at these companies. Testing is still an unsolved problem. Yes, there are large companies everybody talks about, but if you go inside the companies There's still a lot of manual testing, lots of dog fooding. And nobody cares about non-engineering product teams, which I passionately care about. I wrote a bunch of tests, even for our legal and marketing teams at Coursera. So I'm all about like every team needs to have a high quality and deserves it from others. And then why now was simply, so that was why me and why my team. But why now is simply that all software is gonna get 10x crazier with much of it being written. You already see the problems. We saw this earlier. You see issues where people are posting API keys in their vibe-coded app, or they have security issues, or just the UI doesn't work. And the second is, as I mentioned, partially autonomous software. Every company, like literally, I'm looking at some restaurant websites and they're adding MCP and AI chatbots and all that. This is getting crazier And if software becomes like that, testers become extremely important and they need tools to fight it.
[00:08:53] Joe Colantonio I love that. So you're not saying this is replacing anyone. It's actually it almost elevates testers to even being more needed.
[00:09:02] Vasusen Patil Correct. I mean, what's that famous thing people say like in an era when like execution is so easy, all that matters is taste. And I always say, yeah, taste and a good tester has taste. I've worked with some of the most brilliant ones in my life and they're not automation monkeys, right? I am sorry, but like we sometimes get confused. It's not about the automation. That's not what the job is.
[00:09:29] Joe Colantonio 100% agreed. The name of the company is Donobu. I think you have an interesting reason why you call it that. Maybe you have a good start with that to maybe ask a little bit more what you all do and how you help Playwright.
[00:09:44] Vasusen Patil Yeah, for sure. I mean, we are all struggling with how to call the company, to be honest. It's simply stands for do not build. The company's called Donobu and do not build is something I used to repeat as an engineering mantra all the time to teams. Like don't build stuff in house. Try to buy as much as software you can try to partner, try to use open source libraries, because there's a tendency in, especially high quality engineering teams that they want to build a framework internally, including I've seen very good quality teams end up building frameworks and then get stuck because now they cannot use what's happening in the world out there. So that was it. The domain was available and it's just easy enough for people to remember that it stands for Do Not Build. Our logo came from a Japanese macaque. We call it our Chaos Monkey that has had a little too much sake to drink. And my co-founder and I, we just love Japan. So I think that's what it is, a lot of those.
[00:10:56] Joe Colantonio Awesome. Would you consider a true self-healing test in Playwright?
[00:11:00] Vasusen Patil You've had two people recently on the podcast who have really explained this. And that's been the same opinion I had. One was Jason and one was Ben Fellows. And what is a true self-feeling test, right? A true self feeling test is let's mimic the workflow of a tester. You have an end-to-end test. It's running. Let's say it's testing onboarding and now the marketing team or the product team added a new modal that says, Hey, we are launching Blah. And you have to now remove that modal to continue with the onboarding test. What happens? The test fails. This is 95% of how automation breaks in production. All right? When that happens, you got to do something about it. And mostly the test fails, the team, and team blames the QA team, the QA team blames, the marketing team. Everybody starts arguing that you should have checked this. Then you go in there, you simply add one more line to X out the modal, you continue with the test and then you recommit the code. This is not rocket science's workflow. This is something that should be part of how the tests we should take care of. At Donobu, what we do is when that happens, the intent of the test is actually inside the test itself and that's key. That's the only way this works, right? Today what ends up happening is teams just write JavaScript or TypeScript or Python whatever and then they use some random other tool to remember what this test was supposed to be doing. Then they try to reverse engineer what this test is about by that. What we do is we write the intent inside the test and then we let the test fall back to the AI to figure out, hey is the product updated a little bit or has the test or does the test need any updating. If the test needs updates you write the new test and let the QA engineer at that point review the code to see if this is it. That's why I always claim like wake up to PRs instead of broken tests. That's something that appeals to a lot of people.
[00:13:17] Joe Colantonio That's a good tagline. Are there any classes that I should not heal, I guess? because a lot of people like all self-healing is bad because it could be a real issue. Even though you explain what, 80, 90% of failures are due to that aren't real issues anyway.
[00:13:34] Vasusen Patil Yeah, of course. I mean, it shouldn't do things where, and this is where judgment comes in and judiciousness. How well do you write the test case? Really matters. And if you've seen this in your career, I'm sure, it's like the best of the best write really good test cases. That's what they obsess about and figure out. So what should it not do? If it's a true failure, it should fail. Don't try to pigeonhole and self-heal things that don't need to be self-heal. I'll give an example. If my test said something subjective, like make sure that the fall menu on Starbucks has a nice like fall theme going for it, right? If that's the case, now if suddenly there's a terms of service modal coming up and you cannot access that menu, the test should self-heal. But if the menu itself has become summer, the test you'd fail. It should not try to self-heal that and say, no, actually, let's test for summer. Your intent was you were testing for fall.
[00:14:38] Joe Colantonio How do you test that? Can you give me a demo?
[00:14:40] Vasusen Patil Yeah, would love to.
[00:14:42] Joe Colantonio You did mention a demo. I know people would love to see it folks out of listening. Definitely check out the link down below for the video, but we'll talk through what we're seeing as well.
[00:14:51] Vasusen Patil Awesome. Awesome. So we obsessed over how to make this super easy. And we realized the best way to get started is to give people an IDE locally so they don't have to talk to their CTO or legal team, you don't want one of those cloud apps. This is fully local. None of your data is seen by us. It's all entirely your LLM models and stuff. And it's all Playwright. It's just a layer on top of it. And a very gorgeous UI because we have seen that testers never get good UI. I don't know why the tools don't optimize for it. So we obsess over it. Let me start with a template so we can just go from there. And the Starbucks one that I was just talking about, it's the easiest one to drop. So let's say we are testing this, right? Go to Starbucks, assert that the featured menu page has a, and let's use the word autumn because Americans don't use that word, just to show how much AI can understand, right, that's fall, has an autumn season vibe to it. Do the normal test. Go to this town in Vermont, ensure that this is the address that you see. And also look at the map. Sometimes the map-based testing is very hard with canvas elements and stuff like that. These are local, whatever approved models you have, we are gonna use Gemini 2.5 Flash for our browser. I could also do this test manually, but I'll show that later, which is then just a record and play. With this much, let's just get started. That's the intent of the test. You hit Start Autonomous Flow. It's gonna open the local browser as an authoring flow. And this is us. This is Donobu's control panel for me to take over any time. If I were a tester, I wanted to just be like, no, no no. So here you can see that there's a cookie banner that's showing up. We didn't tell our test what to do when a cookie banners shows up. Let me exit out of this browser side. I'm expecting Donobu to agree to that cookie banner. We didn't expect that. The LLM is Gemini 2.5, it decided what to do. It figured out itself visually where is the featured menu page. And now it's gonna look and assert on this page. So it will try to assert like, does it feel like fall or does it be like autumn here based on what it's seen? Right, and this is a visual assertion. Not everything is visual, Ben Fellows actually towards the end of his podcast made great points about how slow, expensive and all of that it is. I think we are showing that this can be pulled months ahead of what people are thinking is the schedule. So like here you can see it's like now doing the other parts of the test. It's going to search for that town is going to see if it can see the map. Sometimes the map is going to fail but it should pass the assertion for the address here. Oh And by the way, when all of this is going on in the background here, Donobu is taking notes. I'm a passionate believer that AI should not be testing your product all the time. That is silly. There's lots of people when they say it's slow and expensive, they assume that we just have these models pointed at web pages and they're just constantly looking. You don't want to add more non-determinism in an already flaky test suite. We use AI mostly to author the test. That's what it's doing here. It's just taking the actions, essentially what you and I would be doing if you were manually, and so like, let me give an example here. Let's go back and let's say I paused it. I just want to see where it is. It asserted that this mount is there. So, let's see. That's it, that's the test. While it's doing that, it's taking all these notes. What it did with the cookie banner, what it with the assertion for Starbucks featured menu page. So let's see what happened to that assertion. It says it's passed. The webpage explicitly says fall favorites meet new arrivals. All right, we didn't say all of these details. We just said, make sure that it looks like autumn for the marketing team. They really care for stuff like this. What we do is then, and we don't use AI to generate code. This is our own engine that then takes the steps that the AI took and writes very deterministic code that you can have full control over and looks clean all the time. This is Playwright code. This is the test. You can see that it encoded the objective as an annotation. So this available to any time the test fails and an AI model needs to look at it. And now these are, to make it slightly less annoying to read, let's remove, we have options on the code generation side. So let's say that element IDs are volatile. Don't take element IDs AI here. This is, it's going to click agree. These are the failovers. This is what most companies call self-healing, which is silly. This is just like we encode the failover once in there. So test is already. Way less brittle than a normal test. And this is something I've been struggling with. That's such an abuse term right now, self-healing. What do we say? Should we say level five self- healing like a Waymo or is it adaptive cruise control like a Hyundai? What is it? So you can see the funniest part. This is where we overload Playwright. Page.visualassert this does not exist as a method natively. And it only comes because instead of importing tests from Playwright, you're importing it from Donobu. That's what I meant by this is a wrapper on top. And then you could see that this is the assertion that's now been encoded as a test. We can edit all of this in a normal file and just ship it in your own GitHub repo. That's how the test will run assertions for some of these. And some assertions are like standard, like look at the URL, look at that text and those. We have some other cool views for like manual testers especially who want to get into automation. We make sure that they can edit their tests and learn how to create automations this way.
[00:21:32] Joe Colantonio Would you call this a test authoring tool powered by AI then?
[00:21:40] Vasusen Patil Correct. Yes. That's exactly how I would describe it. This is like a souped up version of Playwright Recorder because you could also pause what you cannot do in a Playwright recorder. And by the way, let me just show you, if I really end this test, now it's just going to run the test really fast. I don't know. I started two tests. Let's see. I've never tried that.
[00:22:03] Joe Colantonio Can anyone run in parallel like that. Good to see.
[00:22:06] Vasusen Patil Yeah, I hope so, because it's all TypeScript in there. But it's pretty funny. It just might be my computer's memory that will make it slower. But essentially, right now, it's not looking at it. It's just running the test. It's no longer trying to figure out each step, unless I take over again. And I say, here's the way to take over. And when I say take over, on a Playwright recorder, you don't have this concept of just in plain English explaining, actually, here I also want you to do blah and it can do that or I can just take over and start clicking around and it will record my clicks and my inputs and everything just like the Playwright recorder so it's authoring the test with some AI magic as well.
[00:22:57] Joe Colantonio And because automating maps is very difficult normally, like you don't know what to look for, how to navigate it. It knows the code to write to make it more reliable when you run it then when straight Playwright?
[00:23:11] Vasusen Patil Yes. That's the work that has taken the most amount of effort, I would say. There are lots of browser use agents. We were a little early on it, but now there's quite a few of computer use, browser use. What I get annoyed by them a little bit is that they all use very expensive models and are fully visually based. What we do is we do a pretty smart mix of visual as well as looking at some HTML. You cannot look at both fully because you'll just blow budgets and tokens. We stand by it to the point where we say download the tool for free and try it yourself. Like here's for example, we looked at Gemini Flash to author the test. Let me see quickly if I can.
[00:23:56] Joe Colantonio I was going to ask you, why choose Gemini Flash?
[00:24:00] Vasusen Patil Because it's my favorite, cheapest, fastest model. You can choose anything. Whatever is approved, Anthropic, OpenAI, we support Vertex. Our monetization plays that we do this but for enterprises. And that's where we support a lot more of the local versions of LLMs. And what I'm most excited about is next year, this will be whatever LLM model is running locally on your machine to author the test.
[00:24:30] Joe Colantonio Oh, it'll know. Nice. If someone wants to get started with this, then how would they? Is there a paid version? Is it free? Is it open source? What's the current model?
[00:24:45] Vasusen Patil Happy to share. So currently, you just go to Donobu and download it. We have it open for this month and you can just get it, download it, try it yourself. It's completely free except for whatever is your LLM cost. And if you want, reach out to us. We are happy to spin up a key for people like who really cannot find LLM that is prove in their company or for them to use. So that's how it is currently. And then there's the NPM library package, which is currently available on NPM. It's source available, not yet open source, mostly because we are kind of embarrassed of how much code we are ship without looking at it. The plan is to make all of that as an open source layer on top of Playwright. That's how somebody can get started with it. I can share the details of what 160,000 input tokens on Gemini are, but my assumption, based on what I've done these numbers before, is less than 2 cents for authoring this test. That's the way to do it, and then while running, it's frankly become too cheap to for some of these even for visual assertions.
[00:26:14] Joe Colantonio Can you show again that the option where someone could create a test from the flow it created like it looked like you create new tests from the visual understood it correctly. Do you have like a visual representation and then you were able to drag or you're showing something earlier.
[00:26:32] Vasusen Patil Oh, oh, I see, yeah, for an existing test. We had these three ways a created test is shown to a person. One is, my favorite is the simple list view. It just shows all the steps that were taken, what was the outcome of the step, all of that. The code view is obviously the most powerful and we expect you to download. We even give you GitHub actions runner, Playwright configs, and all of those files. If you just download and run this package, there's even a readme involved. You can just run this locally and your boss will suddenly think you're so much more productive than before without realizing what's going on. And then there's this canvas view. It's frankly our worst view yet, but I've heard a lot of people want it like this where they can edit stuff this way or see it this way and remove some steps.
[00:27:29] Joe Colantonio Now, I know we saw Jason show a pre preview of Vibium and he had kind of a view like this already, but it was nowhere near like this. I don't know this was before Vibum, if I went by the time someone listens to this Vibium out, but this is pretty intuitive what you did.
[00:27:48] Vasusen Patil Yeah, correct. I had a really great, almost four hour lunch with Jason afterwards because it was so funny how we've arrived at similar ideas, except I am all in on Playwright and I think that's the way to do it. I think testers love the tool, the tech, and Microsoft's backing is pretty great on it. I think the future is pretty great for testing.
[00:28:19] Joe Colantonio Absolutely. As you work with companies or in your experience, what's the baseline flakiness rate you've seen? How does this help? Or do you have like any numbers like using this? We found that flaky levels go down or whatever.
[00:28:32] Vasusen Patil What we found when new, especially newer startups who are hiding their second QA engineers. That's what I like to say. We are your second QA engineer who works with that. When that happens, we see that they're usually at like a really horrible 10% to 15% flake rate to start with. Because there's just so much movement in the code base. And we aim to get you to under 2%. One thing we had to learn the hard way is many companies actually don't have good quality talent. And so they actually come to us to help even with the strategy of what should be our quality strategy. So we are, oh, you heard it here first, we are launching an FDET. That's what I'm calling it right now as a program modeled after Microsoft's SDET. I'm going to get forward deployed engineers in test and we are actively hiring for those. So that they would be our like strategy people who we can also help companies directly. So 2% is what we aim for. And I'm happy to share some examples. Let me see if I can pull them up here. Let's say this, and this is public, I'll share this in the, oh, sorry, I'm not sharing my screen. All right. Here's a demo GitHub repo that's on our company account, available for people to see which shows what happens when these tests are running, some of the example ones that we ran. When they're running, while we are doing this, I'm just gonna start a new run, just in case. So we get to see hopefully a live demo of it. I have tests that are running where purposely the apps they are testing will break something or the other. They'll change the product. The test has to look at it and update itself. When that happens, let me give you an example of previous runs. Let's say it was running here. We do the reporting entirely in GitHub. So we don't have a cloud version yet. It's entirely running inside NPM. Running in your own cloud environment, running locally, and all the reports are on GitHub and they get piped to Slack. We want to be where test teams already are. We don't want to have them have to learn logging into our platform and take over their test. I also hate that many companies kind of take your tests hostage by saying, they live in our version and then you can never get them without us. So we are trying not to do that.
[00:31:17] Joe Colantonio It's not necessarily vendor locked then. If someone has an existing suite, I mean, they could easily, because it's code, use it. They just miss out on the creation piece, it looks like, right now.
[00:31:30] Vasusen Patil And if they have code and if they just want to start with one test, we say just change one test from saying import test from Playwright to import test for Donobu and you can just get self-healing capability for that one test and see how it goes from there. Here's an example, Joe. I wrote this website. Okay. It's called Unstable Survey. The reason it's called that is this is what vibe coded. Every five minutes, the survey just changes forms. And the test is, can the survey work still? All right, we are not testing the individual form elements, you are actually testing if the surveys work. Like instead of occupation, it will probably ask for educational attainment or something completely different with a different input. So this is an example where when that happened and the survey changed, the test instead of pass fail, it failed, went back to the AI, the AI redid it autonomously the end. Said, yeah, let's update the test. Here's the self-healed version of it. That's what we do for it. And then whenever that happens, we open a pull request. And in that pull request, you get to see the self-healed portion of the test so that your team is always in charge. We don't expect AI to make these changes willingly. Some teams we have seen are open to just saying, just keeping it open for auto-merge, but I think that's a bad idea. You should actually look at what kind of pull requests they need to open. So let me see an example of something I reviewed recently regarding the unstable survey. So here you go. There's a bug in GitHub, I just realized, when you click on this, doesn't take you to the test. Here we go. So when the survey updated, like instead of asking for name, it asked for full name. Instead of asking for your occupation, the field started asking for age group. And we updated the selectors, the comments, and everything because at its core, we knew what the intent of the test is. That's an example of how flakiness.
[00:33:46] Joe Colantonio But how did you do that at runtime then? Like, I understand creation, but like at runtime, how did know what backups to use that if it did change that, that would handle it? Does that make sense?
[00:34:00] Vasusen Patil Oh, that's a good question. That is the core, that is the thing, right? I mean, it's not even that secret. I explained this to Jason as well on how we think this should become standard according to me. We encoded that objective in the test when it was created in English. Now, I'll share a GIF with you as well. But when the test breaks, what ends up happening is in the runtime, you had these modes where it was rerunning or it was figuring it out autonomously. We trigger the runtime autonomous agent to look at the browser again inside the GitHub action and re-go through the test, rewrite new code, and rerun it again to see if now it's working after an update, and then we diff it and send the pull request. And all of that happens inside Github actions runner, so it's not happening on our cloud. That has been the hard part, making it that small that the whole agent can run entirely there. Oh, actually, I should have got the survey changed as we were talking about it. So that's an example for companies.
[00:35:19] Joe Colantonio Perfect. All right. So every time I speak to testers, even at an IRL event, they're worried about their jobs. They think they're going to be replaced by all this autonomous automation. You showed the power of it right now, but you said testers are going to even more valuable than ever. So but we do see this morphing, especially as we head into 2026. I'm always looking for maybe trends that testers should look out for before it becomes critical for them to know. Any thoughts on that?
[00:35:48] Vasusen Patil Yeah. I just don't believe in this like job loss theory. It's completely backwards how people are thinking about it. And I would say it from my own experience, right? When I moved to Silicon Valley, there was one weekend where we were actually moving servers from our office to another office. And as a junior engineer, it was my job to take the servers out of the rack along with my team and we as quickly as possible moved there. And when Rackspace came out and then Cloud and all of that, we were just like, that's it. So few developers are gonna have jobs now. We are done. And what ended up happening if you, for those of you who don't know, is that there is at least 20 times more software jobs out there than there used to be at that point. And I think that's how these things work. I'll give you an example of something that today is a problem. You talk to any marketing team in any of these public companies and you ask them, how much time do you do manual testing? And they say quite a lot. It's actually a huge annoying part of their jobs when they launch a new campaign. How many of them get a QA engineer helping them? Zero. I can tell you zero. Because they asked the CTO, many teams do. And the CTO says, I'm already underwater. My QA team is already struggling to even just keep up. We cannot give you up. What I'm trying to say is that when the baseline of software, like what you can end up doing changes, it's a cost problem today. It's not a quality problem. I actually go to many of my enterprise customers that you don't have a quality program, you have a cost program. Because for this much money, you cannot have this much done. But if for the same amount, now you can have a lot more done, what you start doing is you start fixing all your quality issues, and you look at it from a lens of true quality professional, not an automator. And I think that's the difference. So if I could tell people what to do is just go back to basics, get very good at finding problems, and use all the tooling you have. Don't get annoyed that there's now a typewriter and I will continue using a pen and pencil because I like that world. I used to like actually making server racks happen but I never do that anymore, right? It's just not a world we do. And I think there's, it's pretty magical. So next time you ask in the conference how many people have even tried AI tools or like Donobu, like anything else out there. Like try Playwright MCP, be very good at what are the limitations, you will be surprised how horrible most of it is, once you try.
[00:38:44] Joe Colantonio Okay, before we go, is there one piece of actionable advice you can give to someone to help them with their automation testing efforts, and what's the best way to find or contact you?
[00:38:54] Vasusen Patil Yeah, for sure. I think the best piece of advice, maybe I'm biased here is, I will say go to Donobu, download it, play with it manually if you don't have an LLM or use Playwright MCP if you have an LLM. Use that. Try those things. Best way to contact me is on LinkedIn. I'm linkedin.com/Vasusen it's my first name. I'm on Twitter, it's the same x.com/Vasusen and I'm pretty active and responsive on all of those. Happy to help anybody with anything.
[00:39:27] I've linked to all this awesomeness down below. Thanks again for your automation awesomeness. The links of everything we value we covered in this episode. Head in over to testguild.com/a563. And if the show has helped you in any way, why not rate it and review it in iTunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe, my mission is to help you succeed with creating end-to-end, full-stack automation awesomeness. As always, test everything and keep the good. Cheers.
[00:40:06] Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation, testing, and DevOps. If you haven't yet, join our vibrant community at TestGuild.com where you become part of our elite circle driving innovation, software testing, and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges, we're excited to collaborate. Visit TestGuild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve.
[00:40:49] Oh, the Test Guild Automation Testing podcast. With lutes and lyres, the bards began their song. A tune of knowledge, a melody of code. Through the air it spread, like wildfire through the land. Guiding testers, showing them the secrets to behold.
Sign up to receive email updates
Enter your name and email address below and I'll send you periodic updates about the podcast.