Testing AI: Engineering Confidence in Non-Deterministic Systems with Jason Arbon

About This Episode:

In this episode 600 of the TestGuild Automation Podcast, Joe Colantonio talks with Jason Arbon, founder of Testers.ai, Jank.AI and IcebergQA and author of the new book Testing AI: Engineering Confidence in Non-Deterministic Systems.

Take Our 2027 Survey Now: https://testgld.link/27data

Jason makes a case most testers have not heard yet. Coding is being absorbed by AI. Specification work is thinning out. Product, development, and test roles are converging into one. And when the music stops, the only seat left belongs to the person who can look at what the machine produced and make an evidence backed call on whether it ships. He calls that confidence engineering, and he argues it is not a rebrand of QA. It is what QA was always supposed to be.

Along the way, Joe and Jason get into the containment problem and why alignment, not lockdown, is now the real safety goal. They dig into why testing cost scales quadratically, meaning ten times more generated code creates roughly a hundred times more testing demand. Jason also pushes back hard on skeptics of agentic testing, pointing out that almost nobody has run the obvious experiment of testing a site themselves for a week and comparing their results against what AI finds.

You will also hear Jason’s most practical piece of advice in the whole conversation. If you are not running the same suite five times against the same build and looking at the actual results, not just flake, you are not testing seriously in an AI world.

Plus a detour into grokking, the Chinese Room, and Geoffrey Hinton, because it would not be a Jason Arbon episode without one.

Listen up!

0:00 / 0:00

Join the Guild for (FREE)!

Email New Tab

Exclusive Sponsor

The State of Automation 2027 Survey is open

For over ten years, this survey has tracked what is actually happening in test automation, not what vendors say is happening. It is completely independent, completely vendor neutral, and the results are free for the whole community.

The last decade of data captured the shift to open source, the rise of Playwright, and the arrival of AI in the everyday tester’s workflow. This year’s edition looks at how teams are handling AI on flat budgets, and whether anyone has figured out how to test AI systems themselves.

It takes a few minutes. Your answers make the report sharper for everyone who reads it.

Take the 2027 Survey Now: https://testgld.link/27data

About Jason Arbon

Jason Arbon is the Founder of testers.ai and a Principal at XBOSoft, where he helps organizations transform software quality for the age of AI. He is the author of the new book Testing AI: Engineering Confidence in Non-Deterministic Systems and co-author of How Google Tests Software. Jason has led testing and AI initiatives at Microsoft and Google, and previously founded test.ai, one of the earliest AI-powered software testing companies.

And a life-long fan of Joe Colantonio–and long time listener of the TestGuild. Day one!

Connect with Jason Arbon

• • Company: testers.ai
  • Blog: testers.ai
  • LinkedIn: www.jasonarbon
  • Twitter: www.jarbon

Rate and Review TestGuild

Thanks again for listening to the show. If it has helped you in any way, shape, or form, please share it using the social media buttons you see on the page. Additionally, reviews for the podcast on iTunes are extremely helpful and greatly appreciated! They do matter in the rankings of the show and I read each and every one of them.