Challenge Accepted: Rethinking Automation with Benjamin Bischoff

[00:00:00] In a land of testers, far and wide they journeyed. Seeking answers, seeking skills, seeking a better way. Through the hills they wandered, through treacherous terrain. But then they heard a tale, a podcast they had to obey. Oh, the Test Guild Automation Testing podcast. Guiding testers with automation awesomeness. From ancient realms to modern days, they lead the way. Oh, the Test Guild Automation Testing podcast. With lutes and lyres, the bards began their song. A tune of knowledge, a melody of code. Through the air it spread, like wildfire through the land. Guiding testers, showing them the secrets to behold. Oh, the Test Guild Automation Testing podcast. Guiding testers with automation awesomeness. From ancient realms to modern days, they lead the way. Oh, the Test Guild Automation Testing podcast. Oh, the Test Guild Automation Testing podcast. With lutes and lyres, the bards began their song. A tune of knowledge, a melody of code. Through the air it spread, like wildfire through the land. Guiding testers, showing them the secrets to behold.

[00:01:53] Hey, it's Joe, and welcome to another episode of the Test Guild Automation Podcast. Today, we were talking with Benjamin all about LinkedIn posts he recently had where he made a comment where he said, when it comes to software testing and automation, I disagree with everything, justify me to your podcast and find out yourself. So challenge accepted. We'll also talk about a bunch of other things. He's an API expert, a testing expert in general. Before we get into it though, if you don't know, Benjamin has spent over 15 years as a software developer and trainer. In 2016, he transitioned to test automation after being captivated by S elenium. Currently, he works as a test automation engineer in Germany, focusing on backend and API testing, and Benjamin also authored the book writing API test for Karate. If you haven't gone yet, we'll have a link to it in the show notes for sure. And he also maintains the open source project for Cucumber, BDD, parallel test execution and reporting, so he really knows his stuff. He's also a renowned conference speaker and shares insights on testing, automation, and software craftsmanship on his website Softwaretester.blog. And it's not here, but he is an awesome magician. If you ever see him at a live event, he can does magic on the fly. Very cool. You don't want to miss this episode. Check it out!

[00:03:04] Joe Colantonio Hey Benjamin, welcome back to The Guild.

[00:03:09] Benjamin Bischoff Thank you for having me back.

[00:03:11] Joe Colantonio Awesome.

[00:03:11] Benjamin Bischoff Glad to be here.

[00:03:13] Joe Colantonio Great, great. Yeah, it's been a little over a year. And I had you here in April last year. What you been up to since then? What's new?

[00:03:19] Benjamin Bischoff Oh, I've been doing a lot of API testing, but not only. I mean, recently, in the last month, I've been focusing on rewriting our internal test framework for front end tests, even though I'm more in a backend role now. But I took the chance and a little time that I had off to do that because that was needed.

[00:03:44] Joe Colantonio Awesome. That's a good place to start then. As a magician, though, before we get into it. Do you work on new tricks all the time, or is there anything new that you've been working on?

[00:03:52] Benjamin Bischoff Oh, absolutely. I've been doing a few little shows here and there. Last time was our internal QA meetup that we had, and I always try to integrate some magic, at least magic principles, into the talks. I do also the future talk.

[00:04:09] Joe Colantonio Love it. So you're saying you're rewriting a frontend test framework, were there any surprises you found when you saw this? This is a complete rewrite. This is what made you write that post. You disagree with everything you know.

[00:04:22] Benjamin Bischoff That was actually another trigger to this post or multiple ones. But the test framework I rewrote because we this had been in existence for about eight years. I wrote the one before two, and then we had a lot of learning through constant usage of it. And also technology is evolving and Selenium and Cucumber and all the parts that we're using, also things like JUnit and Maven, significant advancements. So it was just needed to be up to date.

[00:04:58] Joe Colantonio So after doing this, if someone has they're creating a framework now, do you have any tips on how they can make it. So it's easier to upgrade? Because obviously over time you need to extend you need to make changes. Is there anything you could do to create, I guess, a maintainable framework over time?

[00:05:12] Benjamin Bischoff Absolutely. I mean, the first one was maintainable as well, otherwise it wouldn't have lasted eight years.

[00:05:19] Joe Colantonio Right, exactly. Yeah.

[00:05:20] Benjamin Bischoff I mean, we could have also gone using it. The tips I have is stick to clean code principles, read a lot about design patterns, do things like not building everything into a core framework, but also having a plugin architecture to be able to just have on the fly functionality in the framework that you need. So yeah, keep it very modular, but also don't over engineer it. That's not worth it.

[00:05:51] Joe Colantonio Absolutely. Do you recommend someone owns the framework or does like do you have multiple sprint teams that are constantly checking in code. So the framework training not Crud over time, but it's like growing and growing and probably getting bloated. How does that work?

[00:06:05] Benjamin Bischoff For us, we are a pretty big QA team. It is belonging to QA. I, along with two of my colleagues who are also automation engineers, are the core maintainers of it. We definitely keep an eye on the features and the for requests. So we treat it as, not open source but like internal source. People are absolutely invited to contribute to it, but they have to pass us because it's used in multiple projects, and we don't want the framework to become unstable in any way.

[00:06:45] Joe Colantonio Nice. How do you resist the urge of just rewriting it just so you can learn, maybe. So you want to learn Playwright. So like, oh, we're going to rewrite the whole framework and do it all in Playwright or whatever, or WebDriverio or whatever it may be like how do you resist the urge of a lot of times people take this a chance. I'm going to rewrite in the latest and greatest, whatever they may be at that time.

[00:07:03] Benjamin Bischoff It's funny that you mentioned that because my latest talk that it's going to premiere tonight at a meetup is exactly about that. Why it can be better choice to use old tools that you already know and not go with a shiny new things that everybody raves about or hypes about. I think for us, because we have been using those tools already for such a long time, everyone is very familiar with it, so it makes it way easier, way faster to rewrite this stuff in the same kind of technologies, but deal with all the learnings we had outside of the technology, if that makes sense.

[00:07:48] Joe Colantonio Well, I guess you can expand on that. So meaning like a lot of times people use the technology and they think it doesn't do something because they don't know enough about it, and then they create something to do, something that it might already do. I know if that makes sense, but is that what you mean by the learnings?

[00:08:01] Benjamin Bischoff Yeah, but also things like how you use the framework. So thinking about the customers. So our QA engineers who have to write tests with it and create glue code and create components, we just wanted to make this easier for them. This is not really dependent on the technologies we use internally, but more how we expose APIs or ways to create the tests themselves.

[00:08:31] Joe Colantonio All right. So you give an example like how did you make it easier then based on what you learned?

[00:08:36] Benjamin Bischoff For this iteration of the framework, Selenium-based framework, I got rid of all the page component factory and page object factories we had before. I mean, even Simon Stewart, the creator of Selenium, says it might not be the best idea in any case. Also we had pages and components completely separated, so oftentimes we had to implement the same functionality in both. If we wanted to find that element in a page, there were special functions in the page class. If there were components that we wanted to find some elements and we would have to recreate this. But now everything is just one. So a page is a component. Subcomponent is a component. So it's just you can have unlimited hierarchies and everything works exactly the same way. Just little things like that makes just much easier to implement.

[00:09:40] Joe Colantonio Nice. I know probably a year ago Selenium 4 wasn't out. And I think Selenium 4 adds a lot of new features that makes it a lot more modern I guess you could say. Did you do add any new functionality into your framework based on Selenium 4. Was that one of the factors as well?

[00:09:56] Benjamin Bischoff Yeah, it was less about Selenium because we're keeping the cutting edge version of Selenium all the time also in the old framework, but it was more about the other parts like JUnit 5 is now in place and not 4 anymore. Also, we are ready for any future of cucumber in. Before we were stuck at a specific cucumber version because we did some hacks around it and this is not needed anymore. But still, I mean we are experimenting a lot at the moment with Selenium Bidi. I know it's far from being ready and accepted, but still we are doing a lot in this direction to just be ready when it's official.

[00:10:42] Joe Colantonio Once it is official, what do you think? If someone's just having WebDriver Bidi for the first time, what is that going to allow you to do?

[00:10:48] Benjamin Bischoff This is basically allowing us to get messages from the browser directly independently of the Chrome DevTools protocol. At the moment, we can do things like that, like getting specific locks and messages from Chromium-based browsers, but bidi will allow pretty much all the browsers to do that. Having a WebSocket connection right to the browser, not relying on the typical Selenium pulling mechanism to get some insights into what's happening in the browser. But the browser can directly send messages back to Selenium.

[00:11:28] Joe Colantonio Awesome. How did you refactor this without breaking anything? Do you have like a process?

[00:11:35] Benjamin Bischoff I also incorporate my own learnings because I found that one of the pain points of the old framework was documentation and having good examples. I actually started with a demo project. It was like test-driven development, but not on a unit test level, but on the whole framework level. I created this demo. How would you say this demo test project and then implemented the framework functionalities I wanted to showcase. Yeah, basically. I mean, the old test framework is still running in all of our applications, but in parallel, we started implementing the new one and porting tests over. So they are now running at the same time, basically, so we can compare exactly what's happening. And if we reached the feature parity to the old framework and also measure things like performance and speed and resource consumption of both.

[00:12:41] Joe Colantonio All right. I'm gonna move off this topic soon, but I think there's some everyone struggles with it. So after you did the refactor you now have two running in parallel, is there anything you learned based on assumption you may have had before the refactoring like, oh well, that didn't do. I thought it was going to do, or that worked better than I thought I was, and work that worked worse than I thought it was going to work?

[00:12:58] Benjamin Bischoff Actually, worked a lot smoother then I thought the whole rewrite. Now we're hitting some things that we hacked in the old framework that I'm currently thinking about how to do it in the new one. But in general, what surprised me the most when going from JUnit 4 to 5 and all the new versions of the dependencies was the resource consumption, especially when running them in CI/CD systems. It just takes way less resources, especially when running in parallel. One thing I have to mention too, because I forgot one of the main reasons for the rewrite was also that we basically added parallel runs as an afterthought on the old framework, and now we started with parallel runs and built everything around it.

[00:13:53] Joe Colantonio And what allows you to do that as a JUnit 5? What allows you to do the parallel runs better?

[00:13:58] Benjamin Bischoff JUnit 5. Definitely much, much nicer for parallel runs. Also, Cucumber and JUnit 5 work really well together. So yeah. And also, of course, the framework itself is now built to be thread safe. So all the components we have cannot mess each other up anymore.

[00:14:23] Joe Colantonio Nice. So if someone's listening and they're using JUnit 4, do you recommend that they do upgrade to JUnit 5 as soon as possible?

[00:14:29] Benjamin Bischoff I would certainly recommend it. I mean, there's also the JUnit 5. I don't know what it is called legacy or something that allows you to run JUnit 4 tests with JUnit 5. This is already a good step in the right direction, but I'm actually surprised how little it is still used. I mean, a lot of people are still relying on number 4.

[00:14:55] Joe Colantonio Right, right. They're probably afraid. You don't want to touch thing that's working but who knows?

[00:15:02] Benjamin Bischoff It might be. But sometimes it makes sense.

[00:15:04] Joe Colantonio Right, right. All right. I want to dive into a little bit, like I said, that what caught my attention was that LinkedIn post where he said, when it comes to software testing automation, you disagree with everything. So maybe we break that down. Why did you write that? What were you talking about? And I'm just curious to know if I agree with you or disagree with you.

[00:15:23] Benjamin Bischoff If you look at LinkedIn, you see a lot of posts, especially in the spaces of test frameworks and test automation and general QA that are plain wrong, that are misleading, people spread old the information, people compare test approaches and tools that shouldn't be compared. So it's really, really rare that I see a LinkedIn post test related or a software related post that I say, okay, I agree with that, especially when it comes to comparisons of Selenium versus Cypress, Selenium versus Playwright, which is not a comparison that makes any sense. Or just today, I saw a post with the different types of testing where somebody created a graph of how you can test what test approaches there are, and there was manual testing, and then all the different kinds of tests like performance tests and integration tests, all under manual testing. And then there was another branch right besides this that said test automation. This person considered test automation as something completely separate. And also called the other side manual testing, which is always something that triggers me.

[00:16:52] Joe Colantonio Yeah. So that's something that triggers me as well but the opposite is when people rename things. And so when you have a conversation, you pretend like you don't know what manual testing means. But everyone knows a manual testing means. But I get the gist of the concept of not calling a manual or even automation where they call it checks or verifications. But I mean, you know it's automation testing. Why do we pretend? I don't know, anyway. I don't want to get into that but I understand the point. How deep do you get this LinkedIn articles? So a lot of times I'll read an article and people just go by the title like Selenium versus Cypress. And the first, if someone read the article, the first thing would said, clearly you can't really compare Selenium versus Cypress, but this is going to compare whatever the features and whatever or how to decide between tools. You need to know how to decide between the tool. So you need to do some sort of comparison. How do you address it then so that you are able to say, all right, it's not Selenium versus Cypress, but here are some options for automation or here are pros and cons like how would you address that?

[00:17:50] Benjamin Bischoff If there are articles and I have time, I definitely read them. Otherwise, I won't comment on that. I mean, there are people that I will not name on LinkedIn.

[00:18:02] Joe Colantonio Is it me?

[00:18:02] Benjamin Bischoff No, no, no, not you. But like very well-known people of testers that jump on every conversation.

[00:18:12] Joe Colantonio Yes.

[00:18:14] Benjamin Bischoff Every conversation we would see the same. I would say two names, both of them I blocked because of that. And I don't want to be like this. I don't want to just if people spent an hour writing long posts on LinkedIn that I don't agree with, I don't want to say no, you're wrong. Good bye. If I write something, I try to give reasons why I think they might be wrong, but I'm always willing to learn. If people say no, I think he meant it this way or people give me good reasons, then of course, I will change my mind.

[00:18:58] Joe Colantonio What's the best way for people to find out more about good automation and testing? If you think LinkedIn can be kind of a minefield, do you have any resources? There you go. Okay, this go here, you're probably better off. Is it just more like, do your own POC. Can you handle it yourself type deal?

[00:19:15] Benjamin Bischoff That definitely also find people that you trust that are experts in your field. Don't try to just blindly follow people that don't know how to communicate. Read a lot. I mean, read documentation of technologies. Follow websites like yours. Of course, follow your podcasts. Definitely, there are so many good resources in terms of testing. A lot of books. There is Ministry of Testing. There are a lot of very nice communities. I'm a member of the Cucumber and Selenium community, and I'm also a member of the Karate Slack Channels. And so just carefully choose who you talk with and about what.

[00:20:05] Joe Colantonio Yeah. So maybe we'll talk a little more about communication because I may know the people you're talking about. And it's funny, a lot of times people come off really poorly into comments, but when you speak to them in person, you're like, this person's lovely. They make sense, but all sudden you get them in a comment and you're like, oh my gosh, this is horrendous. Why would you write this? But do you have any suggestions or just recommend like not getting into it? Like when you communicate, like you said, just have an open mind. Like any suggestions for someone? That's okay. I just want to come off better when I communicate. I don't come off so harshly and I'm leaving a comment.

[00:20:39] Benjamin Bischoff Yeah, I've definitely fallen into this trap before. Just if I'm in a bad mood and I read something that triggers me, then I can snap to and write something. But if that happens and I find it out later that and think about it, then I will also apologize. But yeah, I don't know what it is with some people that maybe they just think they know it all. And everyone else I don't know, everyone else doesn't. They're like in this ivory tower looking down on everyone and saying, what are they doing?

[00:21:17] Joe Colantonio Yeah. I get in trouble with a lot of people because I'd like to just present information. I have a speaker. Speaker speaks their truth. People listen. I want them to decide. I try not to inject my opinions on it. And sometimes people are like, well, you should have said something or you should have, but it's not my personality to do so. But anyway, that's how I like to approach. I give the information and let people decide on their own where to go.

[00:21:40] Benjamin Bischoff Yeah, that's why I like this podcast.

[00:21:44] Joe Colantonio You did mention Karate. I know Peter, Peter is awesome, but sometimes Peter could come off kind of, Peter is great, but sometimes some of his comments could be taken the wrong way. So a little bit more about Karate if people maybe haven't tried it for whatever reason, I think you have. I don't know if people know Automation Guild, we have monthly sessions. You're going to be doing something on Karate UI testing. I think when people hear Karate, they think API testing and they don't know that has all this other functionality. I guess maybe a quick plug about Karate UI testing.

[00:22:13] Benjamin Bischoff Okay, so yeah, Karate UI is a way to do UI testing using Karate, but with all the benefits that Karate API testing has, like mocking things or authorization and authentication. What's extremely powerful is just mocking APIs while doing UI tests. I will show that. And finally, Peter told me that the UI testing was actually the first part that he did in Karate. And basically when you look at Selenium, it is an API in the background. It's making API calls to the WebDriver. So it is just API communication. That's why it works. And it can be really, really powerful if you combine all the disciplines that Karate has.

[00:23:08] Joe Colantonio So I don't want to put labels on you, but it seems like you're a big Karate advocate. Did you plug this into your work framework because it's just something you like to work on on the side?

[00:23:16] Benjamin Bischoff I did not plug it into our framework. This is always kind of hard decision what you put in a framework and whatnot. For us, we have very specific jobs for the frameworks. So we have the UI part that we use my framework for, and we have API tests and gRPC and GraphQL tests that we use Karate for, because this is just better. I mean, I could potentially build an API test framework myself, but why should I when there is one of the best out there already that is free.

[00:23:55] Joe Colantonio Alright. There's a lesson there, because I would think most people would say, I'm creating a framework, I'm going to create my own API integration. A lot of times they don't realize, let me check out these other solutions that are out there. I mean, I use to do something called Serenity that was like a wrapper on top of a lot of things that made things thousands times easier. When I'm not focusing on coding, I'm just focusing on using a tool to help me test.

[00:24:17] Benjamin Bischoff Yeah, this is the thing. You have to know exactly what tool you use for which job. There is this famous Maslow's hammer that when you have a hammer, all your problems look like nails.

[00:24:29] Joe Colantonio Right, right.

[00:24:31] Benjamin Bischoff Yeah. You need to be very careful. And this way, we can actually have full control over our UI framework and the API stuff that we do in Karate. This is so advanced already that we don't really need to change anything there.

[00:24:50] Joe Colantonio And I guess there's a fallacy. Everyone's always looking for the one tool that does it all and not realize maybe you can have depending on even teams, maybe one team is better suited for Cypress and other team is better suited for Selenium or whatever you can use to get the job done, I guess sometimes.

[00:25:06] Benjamin Bischoff Absolutely, yeah. And I always tell my team if there is anything that is better for our use case for UI tests, then let's use this. I mean, I used to be different. I used to protect my software at all costs and consider it my baby and but not anymore. And it's just about the value of this solution creates. And this is it.

[00:25:32] Joe Colantonio Yeah. It's hard not to fall in love with the solution, though. And sometimes that becomes you like you get in LinkedIn you're like, no, this is the only way, the only tool so. Good, nonetheless, don't fall in love with tools I guess or solutions.

[00:25:44] Benjamin Bischoff Yeah, absolutely. But still try out tools. I mean, I tried out Playwright really like it. Tried out Cypress. I really like it, but for us this just works a little better.

[00:25:55] Joe Colantonio Exactly. Love it. And you have good reasons for it. So that's what I like. I've been showing you a little bit more. And one other thing I want to touch base on, I know we're almost running out of time here. I like to keep this to half hours. You wrote something about. You were doing a hackathon, I think, at work. And you did something with AI and you scored first place. I have to ask you about AI. What are your thoughts on AI? What did you do with AI? What was it?

[00:26:20] Benjamin Bischoff And it was basically a project to improve how we show images on our website and which images we show where. Fortunately, I cannot give you too many details there, but I like using AI tools. I am very critical about what they can and what they can't do. I think they're very much overhyped, especially when you play a lot with different ones, and to realize that they can help you if you know what you're talking about. And if you know what you want to know already, then they can be a super cool rubber duck. I mean, you can give them your problem. They can give you ideas. Oftentimes, that will not work. But when you give them why they not work, they give you better ideas. I treat AI as like a co-developer that I have a conversation with, but I would never trust it to just generate code for me and then put it into production. We're far away from that.

[00:27:29] Joe Colantonio I love that. I wonder if someone has a rubberduck.com that would be a great domain. So yeah, I've been asking people about AI. Is it overhyped, under hyped, and under hyped. And to you seems like it's overhyped at this point. But you're not ignoring it though. Oh it's overhyped. Forget about it.

[00:27:46] Benjamin Bischoff No, no, I think it's overhyped, but it can be pretty helpful. But it's just another tool at this moment.

[00:27:54] Joe Colantonio So if someone's listening and they're like, okay, you mentioned some tools. Do you have any AI tools you recommend people playing where to get familiar if they want to see what it can do and what it can't do?

[00:28:03] Benjamin Bischoff I really like GitHub Copilot because it's also integrated into different Ides like IntelliJ idea and VSCode both work really well. I mean, you just and it learns locally from your code, so it can actually suggest two things in your coding style. So this is really nice. Also, I found that Gemini, the Google one can be quite good if you prompt it well, for like more general things. I also like Microsoft Copilot that the one that is in Bing, but set it to strict mode if you want to know something about coding.

[00:28:48] Joe Colantonio Great advice. Okay, Benjamin, before we go, is the one piece of actionable advice you can give to someone to help them with their automation efforts and what's the best way to find or contact you?

[00:28:57] Benjamin Bischoff Keep on learning. It's my advice for you. Don't fall into the trap of just following one person on LinkedIn and just taking everything they say for granted without questioning it. Find different opinions. I think that is the best way to just get your own opinion. The best way to contact me is via my website softwaretester.blog. It has all the social media stuff in there. So this is a good starting point.

[00:29:28] Thanks again for your automation awesomeness. The links of everything we value we covered in this episode. Head in over to testguild.com/a499. And if the show has helped you in any way, why not rate it and review it in iTunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe, my mission is to help you succeed with creating end-to-end, full-stack automation awesomeness. As always, test everything and keep the good. Cheers.

[00:30:33] Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps, and automation professionals in the world, we'd love to have you join the FAM at Testguild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real-world value that can improve the skills or solve a problem for the Guild community. I love to hear from you head on over to testguild.info And let's make it happen.