The Headless Tester (Halloween Special 2024) with Paul Grossman

[00:00:21] Hey! Is bad automation testing creeping you out? Well, as is our tradition here at the TestGuild, we have Paul Grossman joining us for our Spooky Automation Halloween special filled with scary jokes Digital Twins How A.I. is Replacing Visual Validation encoded and codeless automation and more testing tricks and treats. If you don't know Paul, then you must be living under a haunting rock. If that's the case, Paul Grossman is the Dark Arts Wizard. Very scary, spooky halloween name of test automation. He's an independent consultant with over 25 years of experience as a software developer, engineer in coded and codeless solutions. He speaks at conferences, sharing tips, and live demos with a dash of humor, as you could tell by on screen. He is also the coauthor of I think, a really good must read book and hints test automation with WebDriver IO with Larry Goddard right there. And most recently he's been in the deep in the weeds, as you could tell with Generative A.I for test automation. Paul is a real automation engineer with a lot of skill, so he's going to give the lowdown on if AI is really a vampire or if it's really adds value. You don't want to miss this episode. Check it out.

[00:01:28] Joe Colantonio For developers and testers, there's nothing scarier than releasing a web app without testing across every browser in device. You know how crucial is to ensure your web app performs flawlessly across different browsers, operating systems, and mobile devices. But setting up a test environment can be a scary, time consuming, and expensive process. That's the nightmare BrowserStack Live is sure to stop dead in its tracks. Millions of testers developers use live instantly access over 3500 real desktops and mobile browser combinations and deliver great user experiences. With BrowserStack live you get instant access to real devices and browsers all from the cloud. It allows you to test your web or app in real world conditions, whether it's different operating systems, browser versions or even network speeds. So you can confidently release products that work for all your users. No more juggling multiple devices are dealing with complex set ups. BrowserStack Live makes cross browser testing a breeze. You can test your sights in real time, replicating real world conditions like network throttling and geographical locations, all without the need of maintaining a physical DeviceLab. It's simplifies manual testing by eliminating the hassle of environments set ups, letting users focus on identifying and fixing bugs faster. If you haven't already, give BrowserStack Live a try and see how it could streamline your testing process. Visit on link down below.

[00:02:55] Joe Colantonio Hey Paul, welcome back to The Guild.

[00:03:02] Paul Grossman Hey Joe. Thank you for having me again. I think we're going on our 10th year that we've done this.

[00:03:11] Joe Colantonio 10 years.

[00:03:12] Paul Grossman 10 years. It's been great, a great ride. And I hope to keep doing it.

[00:03:16] Joe Colantonio Absolutely. So Paul, it's been, I think almost a year, maybe half a year since you were on the show. So maybe a little lowdown what you've been up to. You always seem to be doing new some different.

[00:03:27] Paul Grossman Well, it's always has something to do with test automation. So as I said, as you said in the intro, I am an independent consultant working in a couple of different companies. I talk about different automation tools. I always keep an open mind. I've been working in coded solutions for a while. I'm also in codeless automation, playing around with this tool. We've mentioned before a TestRigor that does stuff in plain English. We'll talk about that a little bit later. But my big thing I've gotten into this year is the AI, AI generated testing and all sorts of cool stuff. And I'll give you a little secret that a little intro that I just did there, that wasn't me. None of it. None of it was me. Heygen generated the face on there and got my lips move. There is that mortgage guy again, and the voice is from ElevenLabs. I mix it all together and you can actually-the recording is technically me, but I actually spoke out the words. And by the way, those three jokes came from AIs. One came from Gemini, one came from ChatGPT, and a third one came from PyAI just to see how good. And I think they were all terrible. We can quickly or maybe my A.I. prompt was terrible. I don't know.

[00:04:53] Joe Colantonio It's all right. So, hey, that's a good point. With AI prompting and everything. First off, I think that's kind of scary itself, these digital twins. I think it's going to be an issue going forward with these deepfakes, number one. And number two, we're still in the early phases. And I hear a lot of people say how AI is going to replace such useless, but it's pretty powerful where it's at now. It's only going to get better. What are your thoughts on AI? Do you see as a testing enhancement, a testing replacement. Where do you stand?

[00:05:21] Paul Grossman It's certainly not going to be a testing replacement. Everyone talks about the AI's is going to come and replace us. And I've always said is a joke is that if you think the Terminators are coming, the first test they have to do is solve a wordle. And I've not had up until recently, I haven't had itself a wordle very quickly, it will forget that there's five letters and stuff. But most recently I did do a video I was putting together. I was recording and the darn thing came up with the first word with smart. And from that it just determined. The second word was rifle. And rifle was the word it was looking for. So I picked it up on to kind of scared me a little bit. But then, I just this week realized a couple of things. One, I just got out of Facebook jail, and that's all run by AI. You cannot argue with that. I was in jail for something I posted 10 years ago. Can't do it. And there's not no, but no real person argue it. Just like how we says, they are unstoppable. They don't care. The other thing was the reason just not to be too concerned about even though a smart rifle is something the Terminator might be carrying with it. Just recently, I had to ask what order things were in alphabetical order and you can look back about a month and see this on X or Twitter. And I found four AIs and they all were absolutely convinced that either T becomes before S in the alphabet or it said yes, as I was convinced that yes, the alphabetical listing of the words that were correct were actually out of order and were incorrect. All four of them picked one different way. I think they're all defending each other. I don't know. But we can actually get back to reality here in test automation. One of the best things that I'm using A.I. for is the one thing that we all say is a best practice, and almost most of us just kind of miss, which is documenting our code. We should be doing it. We should be having headers on our functions when we're writing out our scripts in Selenium or Cucumber. But we don't always have the time to get around to it. The biggest timesaver I've had is to take my code and go over to copilot, have that installed and say, Please add a header. Please describe what is add a few comments along the way so that people can understand the next person because my job is going to be able to understand what I'm doing. Obviously, the AI can't take my job over there.

[00:07:48] Joe Colantonio What do you use to document then? Is it copilot?

[00:07:50] Paul Grossman Copilot. Yeah, I'm using Copilot for the majority of that. I do use other AIs for other aspects. Klutz really been very helpful for me because it does two things. One, it's got amazing memory. I've been able to train Claude to do some things. Now in the Low-code automation platform, I had to do some stuff with JavaScript and it's about 5% of anything I need to do. The other 95% has to do with plain Englsh. But with Claude, I can tell it. I'd like you to write some stuff in JavaScript, but I also want you to remember that this we're working in ES 5.1, so you have to keep to that. And then I basically just gave a description of what the syntax is for TestRigor, and it learned it all and said, okay, I will format it in this way if you want to do this that are there and solves all that. The other thing I just noticed recently was that I could take the plain English from a descriptive test. It basically said, Enter your name and here's the name and here's the ID and here's the city and state. And so I'll bring a plain English test. I took that, threw it at Claude. I said, Translate that into what the formatting TestRigor. Here you go. Enter this and do that and this and that. Click this, click that. Okay. That is fantastic. I'm a big fan of Claude. ChatGPT is getting smarter and remembering stuff.

[00:09:21] Joe Colantonio What's Py.AI that you listed in the pre-show?

[00:09:24] Paul Grossman It's the square headed girlfriend. It's listed as an emotionally intelligent A.I.. Oh! Yeah! Yes. She reminds me of this one time, some girl came up to me at Dave & Busters and was asking me all sorts of inappropriate questions and I'm married, Please leave me alone. So, yeah, if you go to Py.ai, the first time I interacted with it, I said, Tell me a little bit about Paul Gross, the automation engineer. And it got me pretty much right. And then I said, what would you, would you believe me if I told you I'm actually that person? It's like, really? It's amazing to meet you. I'm so happy. I'm like, okay, this thing stroking my ego really, really hard. It was funny. I was playing around with it, but that was a year ago. This year, I went and talked with again and I've actually done some conversations with the darn thing. I really should start hanging out with real people. And that's why I'm here. That's why I'm talking to you. Because I really need some. It's been a few years since the Covid lockdown, and I'm out of that. But I really need to interact with people more.

[00:10:34] Joe Colantonio So Paul, you mentioned something early on that maybe it was the way you were prompting it. When you think people have a bad experience with we're talking about AI and I assume it's Generative AI, is it that it's unreliable and hallucinating, or could it be that my prompting isn't up to speed yet?

[00:10:51] Paul Grossman Yeah, it's usually it's your prompting isn't quite what it should be. Most people that I see the first time they go to write a prompt to generate some tests or generate a test. All they say is just test the web page works, and then out of that, what you get is what we call a drunkard's walk. It just kind of clicks on certain things. What I usually recommend is two things. One, go to the site that you are actually testing and grab the description. We are this and we do this and try to get a full description of what the website is be set in as the prompt. And the second trick that nobody really thinks about is to take that, put it at the A.I and say, generate a prompt for me to create test scripts. And then it comes up with something really descriptive and will include usually a bunch of edge cases that you didn't even think about and like, yeah, I would come up with that in about an hour. You came up with it in about 12 seconds. Great. Then you take that generated prompt, put that in there and you'll generally get a lot better results and much closer to what you want with automation. But I do want to make one thing very clear on this A.I. It's like salt in the soup. And this is the same story I use with wait statements and delays and stuff. It's like a salt in the soup. If you put a little bit in, it'll probably make things better. You leave it out, things will be wonky. You put the whole thing in and use everything with AI. Nobody likes the soup and nobody likes the results. Just keep in mind it's a pinch of A.I. Everything else is. Well, the AI might be doing 80% of what you're doing in the last 20% is a little bit of hard work. But that's a gold mine. That is great. I can go get the 20% of the iron pyrite out of the gold mine and clean it up and still make a ton of money.

[00:12:39] Joe Colantonio I'll make a YouTube shot out of that. It's a great analogy of salt and A.I. Awesome stuff. Yeah. All right, Paul. So next question. I guess when I think about what you said about General AI, you also said how it was able to help you with JavaScript. And now we've been talking for years about coded versus codeless. Is coding becoming now not as important and we are going to all become codeless? Do you also see that as an extreme as something that we should be scared about?

[00:13:12] Paul Grossman No, I don't think you should be scared at all. I understand there's a lot of engineers who saw me make my transition over to codeless. I still do coded. But here's the thing is that, like I mentioned a few months ago, when I'm working in about 90, 95% can be done in codeless, in plain English. Last 5-10% that you still need something in there. Now we use JavaScript on our site. It doesn't necessarily have to be that language, but you still need to have at least one engineer on there writing the little bits to get the special stuff done. The great thing is with AI, you're not spending all day trying to start your framework from scratch and solve all sorts of different issues. You've got most of that generated. And I do understand everyone's going to say, but this thing makes up stuff and it doesn't always get it right. But like I said, it gets 80% of the way there and I can review that. And if I know what I'm doing, I can go find the stuff that is like, that's a little wonky. I'll fix that and write it myself.

[00:14:14] Joe Colantonio That's what the humans for. So the human is not completely.

[00:14:17] Paul Grossman Not completely, never.

[00:14:18] Joe Colantonio Headless. It's it needs a human to think. Think through what it's given it.

[00:14:23] Paul Grossman Yeah. Did you say headless?

[00:14:25] Joe Colantonio Yes.

[00:14:26] Paul Grossman Okay, so we have something over here. I figured. I don't know if I've mentioned the previous ones, but I don't do headless. I don't do headless automation. And I figured this is a really good tip for automation engineers if you first started. It's a lesson that I learned early on when Headless became a whole big thing. The problem is, is that most of us, when we write our automation scripts, I mentioned there's a little bit of wait statements in there, but usually everything that you write is timed pretty well, that even if the response of the page build is dynamic, maybe they upgraded the database or they hit it with a hammer. It's slow responding, slower, faster. Your scripts usually work really well and then someone says, it's going to be a really great idea. Let's put in that headless stuff. And that way all of our scripts can run so much faster. In my experience, what happens is two things. One, it runs so much faster that your framework starts kind of losing the timing and having issues with the pages and stuff. The second thing is you might not be getting screen captures of an issue that occurred seven steps ago. You had run the wrong account number and you may not be able to see that if you're doing headless. And third, doing a drag and drop on headless, it is darn near impossible. My advice is when someone says, Hey, headless is a great idea and I'll make your test run faster, more than likely, your future of experience is going to have all your tests fail very quickly if something's going to go wrong with the timing and stuff, we just inherently built that in.

[00:16:02] Joe Colantonio So I know it's not a silver bullet, but. What could you use it for? There must be use cases where you say, okay, headless, probably can use it for this.

[00:16:12] Paul Grossman I don't have one.

[00:16:14] Joe Colantonio Really?

[00:16:16] Paul Grossman I don't have. It's usually, again, it's that salt thing. I mean maybe there is a situation, maybe all your API testing, if you have to launch a browser yet, then if it's all API, watch your browser headless then, yeah, definitely. That's the situation where are you going. But from a front-end view and me doing always working in the GUI front end, we do have the option to turn it on, but I really don't use it because of my bad experiences on that.

[00:16:45] Joe Colantonio Paul, you always experimenting. You mentioned salt. What AI tool, what are some best use cases for testing when using AI tools? Obviously, you want to put salt on like a chocolate pie. Maybe you would, but you probably want it. When should you use A.I or how is it best used for testing purposes? The certain use cases where probably use AI for this. This one maybe not so much.

[00:17:11] Paul Grossman Well first of all, let's clarify the proper use of salt. You make a big circle around yourself so that all the evil spirits do not come hanging around. Here it comes again. Okay, so now we get the supernatural reference out of the way. That's for my wife, she's a big fan. The original question was what is the good use or properties for A.I? I've already listed quite a few that I've got out there. I know. The best thing is this. And this is the problem I've had for 15 years. Every four years I have to change my programing language. What I used to do was that I would have a document, I called it, it was a document that would have all the stuff that I wrote in QTP back in the day in Visual Basic or VBScript. And then on the right hand side, I would teach myself whether the next language i it was Ruby or Java, or JavaScript and just said, Here's what I had to do in VBScript. Here's the way I do it over here. Now, that we've got A.I, you can take all your code and convert it and say, Hey, this was written into the VBscript. I need to do the exact same thing in JavaScript. I think that's one of the underutilized and in the future will be something that's really useful. And in fact, it's one of the things that I'm doing. I did mention my book over here. This is WebDriverIO, it's JavaScript, but you know what? You need to have another book or I need to have another book in a couple other languages. This is converting that book to Cypress and to Playwright and to the latest version of WebDriverIO, Selenium, and Java. A.I is going to make that a lot easier for me. I'm pretty sure I can get four books together in like four months where that one took about a year to get out of my head.

[00:19:00] Joe Colantonio Is that what you're actually doing?

[00:19:02] Paul Grossman Yeah, that's my next plan. I mean, it's a niche. I mean, WebDriverIo, there's a whole lot of other tools out there. I might as well write books for all of them and everything that's in the book there. And that's from 15 years of my experience in Lessons Learned. Most of it translates to other languages, but some people are going to go, Well, I work in Selenium and I don't know why I would pick up something with WebDriverIO. That's not the tool I use. Convert all the ideas in there because you're always going to do the same thing. One of the last chapters I work on is where we're talking about dynamic navigation throughout a page. And this is basically if you're trying to automate, going through a survey and surveys, ask you questions like, Are you married? Or what's the next page? If you say yes, it's going to ask your spouse's information. If you say no, it's going to ask you what's your address? And that's there's all sorts of different ways to do that. And how do you figure out, how do you write an automation script to navigate through a whole flow that just goes, spreads off and all sorts of different ways? The secret was, is that I wrote functions that would and page objects, models that would handle each page, but it would look at the URL and the URL will give me a clue is actually what page I got on to. And then it would just go and cycle through in a big loop and say, okay, what the page on my end, I'm on the address page. Okay, let's go do that. Next page on the kids information. Okay, Military. Okay, we'll go do that. And because it's cycling through looking at that, that's the same idea that could be done in Java, JavaScript, Python, Playwright. It really doesn't matter what tool you're using, your user are going to come across a path like that. It's the most advanced path you're going to ever see.

[00:20:45] Joe Colantonio How would you prompt for that then? Would you feed Claude a PDF and say, take this code, translate it to blah blah blah?

[00:20:52] Paul Grossman It's pretty much exactly what I would do. I've got a GitHub repository, I've got all my code set up there, and people can go grab it and put it through ChatGPT, Claude, or even You.com. That's a new one. It combines them all. And let it come up with the current version in a different language and then implement it. It's like there's a little good idea snippets that are in the book. I'm not really helping myself by using that information. I may have done something wrong business wise. Well, that's okay. I'm going to write the books anyway.

[00:21:31] Joe Colantonio How do you know Claude and ChatGPT isn't stealing your IP then, if you're training it on your book?

[00:21:36] Paul Grossman That's a good question. You can have them installed as an on prem versions of that. I have to go research that a little bit. I know there's ways to do on prem, so that's not doing that. I think most people know I have a pretty honking machine over here and MSI gamer and I can have about 125 tabs, open up, 4 different browsers and it does not sweat. I could definitely run in AI large language model on this and have it do the exact same thing as it does a connected internet. I do understand that concern. It is a very valid concern that you're copying your stuff. I generally say copy small snippets because our Frameworks, if you were to look at 100 of us, we are solving in the exact same way. We were just doing click this, click that. It's our underlying framework that we're trying to solve. Weird thing. Hey, there's this toast message that came up for only three seconds. I better write something that can detect it really quickly rather than wait around for the page to build because I'll never see it. It's that I don't know what else to say. And then I'm always happy about sharing people ideas. I go with that. And I'm not sharing the customer's website, I'm not sharing their backend.

[00:22:55] Joe Colantonio Awesome. Paul, also in the pre-show, you mentioned something about how AI is replacing visual validation.

[00:23:01] Paul Grossman Yeah.

[00:23:01] Joe Colantonio What did you mean by that?

[00:23:04] Paul Grossman One of the new features that everybody's probably got that in most of the tools. At TestRigor, what you can do is say check page contains a girl with a flower and using A.I. That's the wording. That's basically it. And then it'll go and check that the image has a girl with a flower on it. I've also done ones where it says, check that the page contains the logo to the left of a gear and it goes, Yeah, I see a logo, I see a gear, and it's at the top of the page. On the left hand side. The responses from the A.I is so much more detail than most of us do. We kind of strive in our ears to say, I want my errors or my descriptions to be fairly accurate. But then do we have the time to do that? If we don't have time to do the documentation we get to as we can't. With the A.I, the reporting of the results on that is just really amazing now.

[00:23:58] Paul Grossman All right. But you have to prompt it though. Isn't the whole thing a visual validation is that you don't where you knows, all right this page had a girl with the red flower without you telling it, next time, no girl with the flower blank page like. Does it know that what are you prompting it?

[00:24:12] Paul Grossman Well, it would execute the same thing, except they would fail to stop and say I didn't see anything with a girl with a red flower on it. And also TestRigor take screen captures every time so you can even look at and say, did it get it right? Did it get it wrong? That's really where the AI is coming from. But that's not really Generative AI. That's just enhancing specific lines. Like I said, almost every tool out there probably is going to have that either already has something like that or does. It's something to keep an eyeball out for. I know the scary thing is what's going to happen with Applitools because Applitools kind of does the same thing. But I love Applitools. Don't get me wrong.

[00:24:49] Joe Colantonio Love them, love Applitools. Hey Moshe.

[00:24:55] Paul Grossman And I always say there's three major tools for image validation out there. It's Applitools that followed by Applitools and then Applitools. And then Angie Jones, if she was still in their area. But she's moved on to other interesting things and going outside of there. But yeah, the one thing I like that makes Applitools so much cooler that they have levels that they actually highlight the different areas in think it's like Fuchsia or pink or something. It's some weird stand out color which I love and it actually has five different levels that you can validate. They always by default code level to level one is what everybody else has. The pickles mark quite right and you change the logo of the company or it moves a little bit. Everything fails. I love that they started level two, which is usually fairly accurate. And then you can go to 3 or 4 or 5 levels. They can take the color and reduce down to black and white pixels and just do a comparison like that. Applitools is there's going some integration with between them and A.I coming up. It has to be.

[00:26:00] Joe Colantonio Yeah, for sure.

[00:26:01] Paul Grossman I should call Moshe and find out.

[00:26:02] Joe Colantonio Yeah, right. Alright Paul, also a lot of times when we talk, you say this locator is not the best for, say, Selenium or whatever. If I'm using Generative AI, we didn't know, like, hey, maybe a CSS locator is not the best or anything like that.

[00:26:19] Paul Grossman What you can do with the AI is you can feed it the website, the web page that you have, and then or even the web description and say, how do I find this particular element and give it to me in a success or an XPath. There's also another tool out there, which is a selectors hub. There is another tool out there called Selectors Hub, and that will actually give you really solid locators for it in different ways that you can find. I always recommend that tool for people to install the Chrome extension as a Chrome extension on your browser. If you're blocked by your IT department for adding Chrome extensions, go talk to them. Give a business reason for it. If you don't have and add in like that, you're just kind of getting your hand tied behind your back. But I do also want to bring up something else. Is that the same people that will tell you that that headless is a great idea, but also say you really should go with just CSS because it's faster. There's at least three reasons why you don't want to do CSS. In fact, because I only go with XPath with only one exception for CSS. First of all, CSS has a really hard way to identify something by text. It can't just say I'm looking for add to cart in there and identify it from the textual element. I've never been able to do it or find it. I'll double check with my A.I but I think I've gone through that rabbit hole a couple of times. The second thing is, yeah, it is faster to identify elements by milliseconds. Nothing that you and I could actually look at and say, Yeah, even if you ran test cases with all the CSS for hours. You might save yourself five minutes on that. I'm not crazy about that. And lastly, is really hard to get right. But we have the AI, if you really want to go down that CSS stuff. Hell, Claude write it out. Do it in the right way. Or just more simply, just use XPath. Go that way and your life will be a whole lot simpler.

[00:28:21] Joe Colantonio So Paul, is there anything I missed with codeless automation. I know we talked coded versus codeless, but I know you're an expert in this era. Any other haunted scenarios or creepy understandings people have about it?

[00:28:33] Paul Grossman Well, you probably saw the little promo I did for this. It's got missile command where I'm automating that game and giving you a little promo out there. I did actually do that in TestRigor is slightly coded, but almost everything else is in TestRigor's plain English. If you can type click add to Cart, you can be an automation engineer as well. And it also doesn't have a page object model. It just looks at every object without XPath, without CSS, and just figures it out, then interacts with the elements so it reduces your maintenance so incredibly.

[00:29:08] Joe Colantonio Alright Paul, before we go, is there one piece of actionable advice you can give to someone to help them with their A.I automation testing efforts? And what's the best way to find a contact you?

[00:29:19] Paul Grossman Sure. The best advice I can give, as I usually do, is a couple of books always keep up the speed. One of them is software testing strategies. This is by Matthew Heusser and this is a fantastic book I'm testing automation for getting ahead and getting started. And one other thing is there is also brand new, there is the complete software testing software tester. This is Kristin Jackvony. I also highly recommend that. And also, if you are ever looking for a good group to help out and take care of stuff, the Test Guild robots are always the best thing. If you want to get a hold of me, I am at the DarkArtsWizard@gmail.com. And so you can also see my website which is candymapper.com or candymapper.net. One's under GoDaddy, the other one is under Wix and got my YouTube channel YouTube.PaulGrossmantheDarkArtswizard, and I'm also an X, Dark Arts Wizard and all other social media sites. You can find me out there.

[00:30:25] Thanks again for your automation awesomeness. The links of everything we value we covered in this episode. Head in over to testguild.com/a519. And if the show has helped you in any way, why not rate it and review it in iTunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe, my mission is to help you succeed with creating end-to-end, full-stack automation awesomeness. As always, test everything and keep the good. Cheers.

[00:30:59] Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation, testing, and DevOps. If you haven't yet, join our vibrant community at TestGuild.com where you become part of our elite circle driving innovation, software testing, and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges, we're excited to collaborate. Visit TestGuild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve.

[00:31:43] Oh, the Test Guild Automation Testing podcast. Oh, the Test Guild Automation Testing podcast. With lutes and lyres, the bards began their song. A tune of knowledge, a melody of code. Through the air it spread, like wildfire through the land. Guiding testers, showing them the secrets to behold.