Jason Arbon
Okay, good point. If you look around, one thing is you look at studies actual academic things, the AI is on average, this is interesting thing. AI on average is better than most of these radiologists themselves, like literally is and guess how radiologists are using AI, but the AI go through thousands of these things efficiently, and they look at the anomalies and look at ones that are the extrema, right? Just like I'm saying, developers are using AI and how testers should be using it. First, you should really be waiting for the AI to be perfect. By the way, it never will be. And also, beauty is in the eye of the beholder. Some testers think AI is wrong because they're not a very good tester, frankly. Or sometimes they think it's really wonderful because they are not a great experienced tester and they're fine with the basic responses. In terms of, I think what you're getting at is, how do you rely on this stuff that is variable and not always predictable? Again, even if you exclude the human in the loop. The way we do it at testers.ai, and I've had to do this, by the way, not just because it's cool to do, but because I had to control quality, is almost everything that's done is peer reviewed by one agent, by another agent. There's actually like a test manager agent that reviews the work of the other laboring AI agents. Also use multiple LLMs, by they way, to cross check each other. And like for every flow, even though the agent thinks it did well, it double checks it again and says, Does this, here is the intent, here's the actual results, here's consequence, like, is that seem correct, right, to you, so there's like four levels of eval and in checking, because I'm a tester, right? But the funny thing is this, think of the reverse. Every tester thinks like, oh, versus some a perfect ideal world that doesn't exist. Look at your Selenium scripts and Java. When you do a flow, right and you're clicking through an app, for example, and you get to the fourth page and that where you're supposed to be, how do you know that you're there? Guess what everyone does, they grep or they do a regex looking for a string on that page, which will be shopping carts or something, right? So they make sure they got there. There's so little validation in existing test automation, it's embarrassing now. The AI looks at console logs, network activity, screenshot of the app, has the context of the whole flow, and it has kind of the combined intelligence of the planet to a degree embedded in it. And when it looks at that last radiology chart picture right on the screen for the doctor to review. That application that the doctor's gonna look at, it looks at the entire screen. And it can, even if you ask it, if you asked the Selenium developer automation engineer, is this the right page? And guess what? It's a radiology chart for somebody that has a problem, right? Guess what the AI can do? The AI can act, and your script finds a problematic chart. The AI not only verifies holistically that you got there correctly, that the flow made sense, but that screen makes sense for a human to look at. Because it can find UI bugs, and if it's confusing or even difficult to read or has usability issues or anything like that, it finds those. Selenium scripts don't find any of that stuff. But most importantly, it can even give you an assessment on this exact thing you gave me, prompt you gave me, of that radiology image. And you can ask it, do you think there's a problem in here? Does this look like a problematic scan as I was expecting? You can actually get that. You talk about the test team, I know somebody might have worked on a medical product in the past. That your automation never did any of that. There's the imperfections in the radiologists themselves but the interesting thing is that on average, it's faster, delivers far better coverage, far less expensive, and can give you far more validation than we even usually have today in most situations. Even that radiology, because the AI can actually look at that radiological output. Try the latest GPT on a radiology image that you used to, you can't, but if you ran it local, you could. But it's better than any Java Selenium code I've ever seen in my life.