Company: Inflectra Corpoation
LinkedIn: https://www.linkedin.com/in/adamsandman/
Twitter: https://twitter.com/adammarksandman
with Adam Sandman
Company: Inflectra Corpoation
LinkedIn: https://www.linkedin.com/in/adamsandman/
Twitter: https://twitter.com/adammarksandman
How do you ensure software quality when the system you're testing doesn't give the same output twice?
Go to https://links.testguild.com/inflectra and start your free 30-day trial, no credit card, no contract required.
That's the core challenge facing every QA team building or testing AI-powered applications today and it's breaking all the rules we've relied on for decades.
In this episode of the TestGuild Automation Podcast, I sit down with Adam Sandman, co-founder of Inflectra, to get into what non-deterministic AI testing actually means in practice, why traditional pass/fail testing no longer cuts it, and what quality professionals need to do differently right now.
We cover:
If you're a tester, QA manager, or automation engineer trying to figure out how to keep up with AI-driven development without losing your mind — or your job — this one's for you.
This episode is brought to you by Inflectra: makers of SpiraTest, Rapise, and the full Spira platform for test management and software development. Real talk: if you're still piecing together your test management with spreadsheets and hope, it's time to check these guys out.
They've been at this since 2006, they've got 80,000+ daily users across 82 countries, and they back every subscription with free support.
Go to https://links.testguild.com/inflectra and start your free 30-day trial, no credit card, no contract required.
AI has significantly lowered the barrier to entry for software development, allowing teams to produce up to 6x more functionality For QA professionals, this creates a dual challenge: you must now test a massive volume of traditional (deterministic) code generated by AI, while simultaneously figuring out how to test new, unpredictable (non-deterministic) AI features, like agentic workflows and chatbots
In a traditional deterministic system, providing the exact same environment and test data will yield the exact same repeatable answer every time In a non-deterministic system, the exact same input can result in a different answer every time Because 100% testability is impossible in these environments, QA must shift from traditional pass/fail metrics to identifying the “acceptable level of risk" for specific use case.
Testing non-deterministic systems is closer to performance or load testing than traditional functional testing
It requires deploying automated input agents at scale to simulate millions of permutations, and output agents to analyze the results against specific parameters (e.g., helpfulness, safety, hate speech) Inflectra is actively developing a tool called SureWire specifically designed to test these AI systems at scale and provide statistical outputs on performance criteria
Yes. AI acts as “institutional memory" by ingesting entire codebases, old test requirements, and version histories. It serves as a dispassionate judge that can explain why old code was written a certain way, assist in rewriting unit tests, and safely refactor outdated systems (like moving from IE11 requirements to modern web tech) without the fear of breaking unknown dependencies
Start by targeting the pain points. Use AI to automate repetitive, boring tasks—like fixing brittle Selenium scripts—to give testers hours back in their day. Once the team sees that AI saves them time rather than adding to their workload, they will be much more receptive to learning advanced skills, such as testing non-deterministic systems or writing requirements
Adam’s final advice to testers is to be courageous and start tinkering. Install tools like Visual Studio Code or Cursor, play around with open-source options, and don't be afraid to break things. Embracing failure during this experimental phase is exactly how new innovations—and tools like SureWire—are born
Keep listening

Automation Testing Podcast · Ep. 612
Robot Framework Test Automation: Automate Web, Desktop, Mobile, and Embedded Systems
What makes Robot Framework flexible enough to test medical devices, low code bots, monitoring systems, and embedded
Nov 17

Automation Testing Podcast · Ep. 611
Playwright at Scale: Run 500 Tests in Minutes with Oliver Stenbom
with Oliver Stenbom
Is your Playwright suite slowing you down instead of speeding you up? In this episode, Oliver Stenbom,
Nov 10

Automation Testing Podcast · Ep. 610
How to Prove Your Release Is Ready When AI Writes the Code
How to Prove Your Release Is Ready When AI Writes the Code? Can you prove your software is ready to ship, or does that answer still depend on a meeting and a spreadsheet? Want to see if your team can prove a release is ready? Book a custom LoopIQ demo with Ashwin: https://testgld.link/loopiqdemo In this episode,
Nov 3

Automation Testing Podcast · Ep. 609
Why Your AI Tests Pass When Your Features Fail with Donovan Brady
with Donovan Brady
Why Your AI Tests Pass When Your Features Fail? Donovan Brady of Leapwork shows how to make
Oct 27