LLM Testing. Free Test Tools, AI Test Management and More TGNS174

[00:00:00] Joe Colantonio What are the 4 top free new tools every tester should check out? How can AI powered test management tools help QA teams balance rapid software delivery and quality assurance? And is your traditional test strategy falling apart with large language models? Find out in this episode of the Test Guild News Show for the week of November 9th. Grab your favorite cup of coffee or tea and let's do this.

[00:00:21] Hey, before we get into the news, I want to thank this week's sponsor, Zaptest AI, an AI driven platform that can Help you. Supercharge your automation efforts. It's really cool because their intelligent co-pilot generates optimized code snippets while their planned studio can help you effortlessly streamline your test case management. And what's even better is you can experience the power of AI in action with their risk-free six-month proof of concept featuring a dedicated ZAP expert at no upfront costs. Unlock unparallel efficiency and ROI in your testing process. Don't wait, schedule your demo now and see how it can help you improve your test automation efforts using the link down below.

[00:01:02] Joe Colantonio All right, so let's kick things off with four powerful tools you definitely should check out. I just dropped a new podcast episode where I go over the top free tools testers need to try and they're all available on Testguild.com. And I go through a full breakdown of each tool, but let's check out each one right now really quick on the website. The first one is the Test Guild tool matcher. This helps users quickly find the right testing framework or solution by answering a few targeted questions. No bias, no endless research. The next one is the Test Risk Calculator, which gives managers and leads a way to identify and prioritize high risk areas in their testing tool pipeline using data driven scoring.

[00:01:43] Joe Colantonio The third, which is one of my favorites, is the Accessibility Scanner, which makes it easy to check out any website for accessibility gaps in seconds, offering plain language explanations and quick visual feedback to catch common compliance issues early. And last is our automation assessment quiz, which helps teams benchmark their automation maturity, highlight where they stand and where they can improve next.

[00:02:06] Joe Colantonio Together, I created these tools for the testing community for a clear picture of your testing overall health from to selection to risk, accessibility, and overall maturity, all without a price tag. So definitely check them out down below. We'd love to hear your thoughts and how I can improve them as well.

[00:02:22] Joe Colantonio All right. Here's a company I haven't heard from in a while. Glad to see they're still round? And it's SauceLabs that just introduced a new tool for AI for Insights. And they're calling this Sauce AI for Insights, which is designed to transform how engineering teams handle test data. It was launched last week and this tool addresses the overwhelming volume of test data that often slows down release cycles. And this is something we see at The Guild all the time people always struggle with test data and according to Sauce Labs CEO Prince, the problem is in generating test data, but interpreting it, which has become specialized knowledge. And this new AI solution aims to democratize quality intelligence by providing instant answers to natural language queries about test results. The company reports that teams using this beta version have seen dramatic improvements, including 99% faster identification, according to them of root cause and debugging time reduced from hours to minutes. The system provides real-time analytics, rule-based insights, and visual outputs for charts and clickable links to relevant test artifacts.

[00:03:24] Joe Colantonio Next up is our webinar of the week. If you're looking to get hands on with AI in a testing workflow, BrowserStack's upcoming webinar with The Test Guild might give you practical insights that you can use right away. So in this webinar, Raul is joining us to go over how AI power test management can help QA teams balance rapid software delivery with quality assurance. It's going to go through many things, but one of them is going to demonstrate how Browserstack's test management platform uses AI agents to enhance the entire testing workflow from planning and creation to execution, maintenance and validation. And they tell me that their AI solution is built with true AI engineering for context and accuracy rather than simple GPT wrappers. You definitely want to attend live to see some demos of AI applications and testing workflows and learn how real time analytics can help your team identify issues earlier and improve testing decisions. And as always, I highly recommend you join us using that link down below. Register now, even if you can't make it, make sure to register down below because you'll receive a recording of the event in a day or so.

[00:04:25] Joe Colantonio All right, so now let's zoom up from tools to strategy because sometimes the biggest testing challenges isn't technical at all. It's usually soft skills or how you can help lead change in your organizations. David Ingraham explores a crucial perspective shift for testing professionals, In this article, Test Leadership as change management. David goes over how testing problems are really about testing itself, but about managing change. QA leaders don't simply evaluate software, they assess how the team adapts to new processes, tools, and methodologies. And this insight becomes really relevant, as you know, all this AI technologies that's coming out there is being implemented into our pipelines. And according to the article, every QA initiative is fundamentally a change initiative, whether introducing automation, shift in quality left, or redefining what done means, these are asking people to alter their behavior and sometimes their professional identities. And David references two different models to illustrate how teams experience predictable emotional reactions during transformation from denial to acceptance. The article also emphasizes that resistance should be treated as valuable feedback rather than opposition. And for effective change management, David recommends involving team members early starting with pilot projects, normalizing iterations, maintaining transparent communication and positioning a new technology as a partner rather than replacement. Really great point, and you can read more about it using the links down below.

[00:05:50] Joe Colantonio And as AI becomes central to modern applications, your traditional testing approaches might be falling short. And here's why testing LLMs requires a completely different mindset. What is it? Let's check out this next article from Awesome Testing, which outlines how traditional quality assurance practices must evolve when testing large language model applications. The author explains that LLM have fundamentally changed testing by introducing non deterministic outputs that regardless of what you do, the same prompt can often produce different responses. And the article introduces a three pillar approach to LLM testing. First is test deterministic parts of the system with mocked LLM components. Second is implementing end-to-end evaluations using benchmark tests that measure business value rather than technical correctness. And the third is establishing robust testing in production through user feedback loops, A/B testing and comprehensive observability. And for a practical implementation, the article demonstrates how to use Wireshark to create deterministic mocks of LLM APIs enabling traditional pass/fail testing for the application's core functionality. It then explains how LLMs as a judge can evaluate system outputs on criteria like relevant and tone by scoring responses from zero to one. And the author recommends tools like OpenAI Evals, promptfoo, and TruLens for benchmarking LLMS applications. Emphasizing that quality becomes a continuous measurement rather than a binary state.

[00:07:17] Joe Colantonio All right. I always felt that being able to properly test something, I need to know how to do it myself. That's just me. That's my approach. I found this next article, how to create your first AI agent. And this by Debbie O'Brien, who's the leading development advocate for Playwright at Microsoft. She shows how to build a simple AI agent using n8n with chat based triggers, then wired to practical services. And in the demo, the agent answers questions like today's weather, current YouTube subscriber accounts, a bunch of other things, as well as email the summary of the conversation.

[00:07:47] Joe Colantonio All right. Whenever we talk about self-healing, a lot of times we were thinking functional automation. But how about if it's for DevOps? Well, this next tool just might help you, Komodor has announced new autonomous self-healing and cost optimization capabilities for its AI SRE platform designed to simplify operations for teams managing Kubernetes environments at scale. The platform uses agentic AI technology that can automatically detect, investigate, and remediate issues in cloud native infrastructure. And this new autonomous capabilities addresses growing operational complexity as industry research shows 88% of test leaders for rising stack complexity.

[00:08:24] Joe Colantonio And last up is the growing opponents of not only automation and functional testing, but also security testing. And one way I gage this is by acquisition. So here's a Bugcrowd just acquired Mayhem Security. If you don't know, Mayhem Security is an AI offensive security pioneer founded by elite hackers, and this move aims to create a comprehensive security testing platform that combines AI-driven automation with human expertise. And this integration will provide customers with automated protection during deployment and human-driven adversarial testing of deployed software. And Mayhem security brings specialized AI capabilities for API testing, code security, Dynamic S-bombs, and reinforcement learning.

[00:09:06] All right, for links of everything of value we covered in this news episode, head on over to those links in the comments down below. That's it for this episode of The Test Guild News show. I'm Joe. My mission is to help you succeed in creating end to end full stack pipeline automation awesomeness. And as always, test everything and keep the good. Cheers.