AI Testing Reality Check: What 13 Founders Refuse to Automate with Joe Colantonio

About This Episode:

I asked thirteen people inside AI testing one question: can AI test your software for you? Founders, CTOs, project leads, and quality leaders, many of them selling AI testing tools right now. Not one said yes.

This episode is my session from the Testear.la 2026 conference, built entirely from twelve months of interviews on this show with the people actually building this technology, including Applitools, Maestro, Bright Security, QA.tech, Cucumber, Ito, and the lead of the Selenium project.

In it you’ll learn why one AI testing vendor deliberately removed AI from the part where tests run, the real productivity number teams are measuring versus the number in the case studies, the four incompatible architectural bets vendors are making and how to tell which one your tool has chosen for you, why AI generated code is producing vulnerabilities far faster than it produces code, and the one architecture rule that stops an agent from editing the test that was supposed to catch it.

You’ll also get the list. Every task these thirteen people said they would never delegate, assembled from conversations that had no coordination between them. Plus two tests you can run on Monday, both free, that tell you whether you’re ready to trust an agent with your suite.

If you’re a tester, automation engineer, or QA leader trying to figure out where you fit in all of this, this one is for you. The people building AI testing don’t trust it to test on its own. That gap is your job.

0:00 / 0:00

Join the Guild for (FREE)!

Email New Tab

Exclusive Sponsor

Quick note before we get into this one.

Everything you’re about to hear points at the same conclusion. The work is moving from writing tests to supervising the things that write them. That’s a different skill set, and most of us are picking it up by accident.

Automation Guild ’27 is where you pick it up on purpose. Year eleven. February 8th through the 12th. Five days, live, with the best E2E automation practitioners working today, and you keep lifetime access to every session.

The early bird list is open right now, and that’s where the biggest discount goes.

Head to testguild.com and look for Automation Guild ’27 early bird, or grab the link in the show notes: https://testgld.link/yearlybird

About Joe Colantonio

Joe Colantonio has over 25 years of experience in software automation testing and is the founder of Test Guild – a dedicated independent resource for actionable and real-world technical advice from industry experts (blog, video tutorials, podcasts, and online conferences) to help improve your DevOps automation, performance, and security testing efforts. He is the host of the longest-running podcast on automation testing. He has interviewed over 400 industry leaders and is the creator of Automation Guild, the premium, annual online event for test automation engineers. Joe has also hosted over 500 hours of live online events for some of the biggest companies in the software testing space. Joe’s company, Test Guild, also provides services to software companies to help them reach a community of highly engaged, growth-oriented, and ambitious audience of 27k+ software testing professionals.

Connect with Joe Colantonio

• • Company: testguild  www.testguild.com
  • Blog: www.testguild.com
  • LinkedIn: www.joecolantonio
  • Twitter: www.joecolantonio
  • YouTube: www.joecolantonio

Rate and Review TestGuild

Thanks again for listening to the show. If it has helped you in any way, shape, or form, please share it using the social media buttons you see on the page. Additionally, reviews for the podcast on iTunes are extremely helpful and greatly appreciated! They do matter in the rankings of the show and I read each and every one of them.