Why Your AI Tests Pass When Your Features Fail with Donovan Brady

About This Episode:

Why Your AI Tests Pass When Your Features Fail? Donovan Brady of Leapwork shows how to make AI tests traceable, cheaper to run, and safe to self-heal.

👉 Try Leapwork Play (free tier): https://testgld.link/podcastsleapwork

AI can write hundreds of tests in minutes. But can you trust what it built? In this TestGuild episode, Joe Colantonio talks with Donovan Brady, VP of Developer Relations at Leapwork, about the biggest problem in AI testing right now: trust.

Donovan shares what he’s seeing across industries, from bleeding edge teams running fully agentic development to financial services and healthcare companies that run on technology but aren’t tech companies. Then he demos Leapwork Play, which turns your requirements, Jira stories, source code, and design files into a knowledge base, generates standard Playwright tests, and traces every step back to the exact line of the requirement it came from.

You’ll learn:

• Why AI self-healing can rewrite tests to pass instead of catching real bugs
• How to trace every AI generated test back to a requirement
• Why generating Playwright once and running it deterministically saves time and tokens
• How to find coverage gaps, prune redundant tests, and run only the tests tied to a code change
• How to control self-healing so it never masks a real regression
• How to run tests headlessly through an MCP server from Claude Code, Codex, or Cursor
• Why exportable Playwright code means no vendor lock-in

Donovan’s key lesson: when his team used Playwright plus Claude, self-healing sometimes rewrote tests to pass rather than to test the real outcome. His advice is to give your AI the full context of the business workflow you’re validating, end to end.

What’s your experience with AI generated tests? Have you caught one passing when it shouldn’t? Tell us in the comments.

#Playwright #AITesting #TestAutomation

0:00 / 0:00

Join the Guild for (FREE)!

Email New Tab

About This Week’s Sponsor: Leapwork Play

Getting AI to write Playwright tests is the easy part. Trusting those tests, and managing them once they’re running across dozens of pipelines, is where teams get stuck.

That’s the problem Leapwork Play is built to solve. Play is an AI-native platform that generates, manages, and runs Playwright tests, built for teams that want the speed of AI with deterministic results they can trust.

You can work with it however your team works. Prompt it in natural language, run it headlessly through MCP, or trigger tests straight from your CI jobs through the API. Everything is built into the platform, including cloud hosted browsers, audit logs, and evidence trails that show what ran and where. No extra third party tools needed.

Leapwork Play went GA on September 30th, and there’s a free tier, so you can try it yourself today 👉 https://testgld.link/podcastsleapwork

Tell them Joe sent you 🙂

About Donovan Brady

Donovan Brady is VP of Developer Relations at Leapwork, where he focuses on how AI is changing software development, quality engineering, and the systems organizations use to validate software at scale. A former software engineer, Donovan spent more than a decade building and leading Solutions Architecture practices across cloud, data, AI, and mission-critical applications. His current work explores agentic software development, Continuous Validation, and the changing role of quality as AI accelerates software delivery. Donovan is a frequent conference, webinar, and podcast speaker and has presented at events including AWS re:Invent, AWS re:Inforce, Microsoft Ignite, StarWEST, and several other marquee events

Connect with Donovan Brady

• • Company: www.leapwork  
  • LinkedIn: www.the-donovan-brady

Rate and Review TestGuild

Thanks again for listening to the show. If it has helped you in any way, shape, or form, please share it using the social media buttons you see on the page. Additionally, reviews for the podcast on iTunes are extremely helpful and greatly appreciated! They do matter in the rankings of the show and I read each and every one of them.