About This Episode:
Are you considering using Playwright for your next automation project? In this episode, Sonja Leaf, Director of Engineering at Cloudburst, will share why her team chose Playwright and how it has worked for them so far. Discover why they moved on from Cucumberjs, some newer features of Playwright, visual test automation, and much more. Listen up!
Exclusive Sponsor
The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs. Try it for free today!
About Sonja Quartz Leaf
Sonja Q. Leaf is the Director of Engineering at Cloudburst, SBC, and owner of SQL.media, LLC. She puts the Q in quality, with her background in structured query language, complex systems architecture, site reliability, browser automation, video game testing, and media publishing.
Since 2006, she has contributed to software releases ranging from commerce solutions and rewards systems to desktop, native, web, and console software. Sonja started helping entrepreneurs begin their own tech companies when she joined Cloudburst, SBC in 2020. Sonja works from her home studio in Minneapolis, Minnesota.
Connect with Sonja Quartz Leaf
- Company: www.cloudburststudio.com
- LinkedIn: aleaf
- Twitter: @SonjaQQL
Full Transcript Sonja Quartz Leaf
[00:00:02] Intro Welcome to the Test Guild Automation Podcast, where we all get together to learn more about automation and software testing, with your host, Joe Colantonio.
[00:00:17] Joe Colantonio Hey, it's Joe, and welcome to another episode of the Test Guild Automation Podcast. Today we'll be speaking with Sonja, all about Playwright Cucumber and all other types of automation awesomeness. We don't know, Sonja is the director of engineering at Cloudburst, SBC, and owner of SQL.media LLC. She puts the Q in quality with the background and structured query language, complex system architecture, site reliability, browser automation, video game testing, and media publishing. And she's an awesome musician as well. Since 2006, she has contributed to software releases, ranging from commercial solutions and rewards systems. The desktop, native, web, and console software and Sonja started helping entrepreneurs begin their own tech companies when she joined Cloudburst in 2020 and Sonja works from her home in Minnesota. I'm really excited to have her. You don't want to miss this episode. Check it out.
[00:01:11] Intro The TestGuild automation podcast is sponsored by the fantastic folks at SauceLabs. Their cloud-based test platform, that helps ensure, you can develop with confidence at every step, from code to deployment to every framework, browser, OS, mobile device, and API. Get a free trial, visit testguildcom.kinsta.cloud/saucelabs and click on the exclusive sponsor's section to try it for free today. Check it out.
[00:01:40] Joe Colantonio Hey, Sonja. Welcome to the Guild.
[00:01:45] Sonja Q. Leaf Hey, Joe. Great to be here.
[00:01:48] Joe Colantonio Awesome, great to have you. So it's been a while you actually spoke at our Automation Guild 2021 event on Playwright in Cucumber. So I thought we'd dive in a little bit around that. But before we get into it, is there anything in your bio that I missed that you want the Guild to know more about?
[00:02:04] Sonja Q. Leaf No, I think you, I think you hit it on the head.
[00:02:07] Joe Colantonio All right, so this is already starting off script here, and that is, I noticed, you know when you started, you were, I think, a quality engineer and it looks like you made a huge leap. And now, the director of engineering. So I just need to know what's that like? How did that happen and what's the role change? If you could dive in a little that'd be awesome.
[00:02:27] Sonja Q. Leaf Sure. I've been making software for a long time. I joined a young company as a senior quality professional. I wanted to be their quality engineer and they liked how I led their practice. I'd had experience doing leadership elsewhere in terms of tech. As our company grew, we decided we needed somebody to help wrangle the engineers and they chose me, the great position to be in. It's been fun to influence quality at this level rather than after the code had been written.
[00:02:56] Joe Colantonio Absolutely. So how are you doing that? How do you have any tips for people? Because a lot of times when you step into a culture like that, it's totally different. You know, the director of the has to do it rather than someone that's actually implementing it.
[00:03:07] Sonja Q. Leaf Well, it helps that I'm already familiar with their products. I'm not just coming in at the leadership level and I know how our code works. I've been an engineer, I've made software, I've tested software and I like to be in the testing land. But really Playwright was actually one of the things that helped me, helped our team become successful. We've had pretty wide adaption and folks are making better, more reliable, more consistent software out of that. I showed it to a couple of my teammates and they went, Oh, that's cool. And there are some other things we'll get into with it that, that are really shiny about some of the latest features.
[00:03:42] Joe Colantonio That's actually what's going to be my next question is when you spoke at the event, you were using CucumberJS in Playwright. So I want to know is, do you still using it? And it sounds like you are using it and it's actually been successful?
[00:03:54] Sonja Q. Leaf We focus just on Playwright and Playwright tests. We do that because we do typescript everywhere else. And Playwright has started to package their own test runner, which has a lot of the same functionality Cucumber had. It doesn't have the easy extrapolation of users stories to put in front of my customer, but I can still get all that information out in my test run, and it was much easier to get my test or my engineers to adapt browser automation. When I said, hey, just write your test, you can call it your steps right here. What's actually happening in one file, rather than jumbling the handful of files that are required to make Cucumber run?
[00:04:34] Speaker 2 Of course, it seems like it's more streamlined, do you still get the readability than you would get out of using BDD.
[00:04:41] Sonja Q. Leaf It is. You do not get the readability, but we found that most of those stories actually end up live in our issue tracking system. They don't end up in our code base. There's, we tried having it in our code base for a while and it was good, except it was hard to keep up with were a small shop, but we move fast. It was easy to include it all in new feature sets, just having it as part of the unit test set.
[00:05:04] Joe Colantonio Nice. So when you see the developers, these are testers, the developers actually writing UI automation and other types of automation?
[00:05:13] Sonja Q. Leaf Yup. We've got a small test team of two and then I've got a handful of developers and there, we're writing our own test code and doing our share manual testing too. We're still doing rapid feature development and there's stuff that even though developers write their own tests, testers think differently. I think you and I know this pretty well, and I've got a good crew that makes sure that the developers are actually building solid code.
[00:05:39] Joe Colantonio Nice. So sounds like you're a smaller team and you probably release a lot than I assume so. You mentioned manual testing, how much manual testing is involved, and how, how often do you release software?
[00:05:49] Sonja Q. Leaf We've got any number of projects going on at a given time, but there's probably every sprint when we're on a two-week sprint. There are two to three releases that go out with a team of probably eight developers, two testers, plus myself.
[00:06:06] Joe Colantonio So, I'm going to dive into the session, maybe the people I met that missed the session you did at Automation Guild why did you choose tooling? But I'm still on this track now of when you started, when we talked to some like, you just started with the Playwright. And now, you know, it's been almost a year. Curious to know, how much your test week grow by, and do you have any pruning in place? Like, did you make any decisions that you wish you did differently when you started than if you were to start now?
[00:06:32] Sonja Q. Leaf You know, Playwright has grown so much over the last year, we've just grown with it. One of the things they've added is visual comparisons. I used to rely on external third-party tools or like there's a JavaScript tool I've used in the past called BackstopJS, a good open-source product. But now that's all built-in. Hillary is one of my engineers. They're the reason I actually know so much about the latest and greatest of Playwright because they've gotten super excited about the release notes that have come out. And as soon as they offered parallel tests, that information was in front of my face thanks to Hillary. So shout out to Hillary.
[00:07:07] Joe Colantonio Now it seemed to be going like gangbusters over there, Playwright actually, I did an event for Applitools called The Future of Testing, and one of the lead engineers is on that call. In his session, they came out with some really slick functionality. And so let's go back in time. When you started, there wasn't this slick, and it's like you still were behind Playwrights. So why? Why initially did you choose Playwright?
[00:07:29] Sonja Q. Leaf Initially, I chose Playwright because I needed to write tests with multiple user interactions. I looked at Cypress. I looked at a Protractor. To play nice in our frameworks. But Playwright allowed me to create a browser context, and then I could create another browser context, and they had their own unique caches and cookies, so I could have. Maybe I have an administrator, user, and an editorial user, and I needed those to interact and rather than tear down a browser, store the cookies, restart the browsers, stash to create a new user. I could do them simultaneously, so my test writing got faster, and running got faster. I didn't also, I don't know you've written your share Selenium. It's hard to wait for elements. And Playwright has that figured out, there's this fantastic method called “wait for Selector”. “Wait for Selector” everything goes through and it is fantastic. Figuring out how to get a dynamic app to click at the right time used to be one of the biggest challenges I experienced, and I haven't had that with Playwright.
[00:08:35] Joe Colantonio Wow, so it's been almost a year since we talked. Still, no issues? Because you mentioned that as well at the event. That was one of the killer features,
[00:08:43] Sonja Q. Leaf Yup. Still no issues. And, I actually, I won't say that. Like, let's say I have an intake form that has multiple questions that are yes or no. Sometimes if I want a quick yes, it's hard to find which yes, I meant even though there's the only one that's visible, there might be many radioelements that are in the DOM, but they've got this new concept that's come out called the locator API, which normally when you write a selector and you select an element and you identify it as a variable, it's got that copy, that specific one, it located that moment in time, but locator, the locator API. You write it just like a selector. But every time you say you do click, it goes and looks for the newest one. So that locator API helped solve some of those item consistency issues that I hadn't seen many of. I was mostly just what I had buried form fields, but the locator API has solved that for me.
[00:09:37] Joe Colantonio Nice, so how's the speed when it runs the test? Is this slower than Selenium in your experience, just as fast as it all depends, I guess, how you develop it, the script?
[00:09:46] Sonja Q. Leaf A lot of it is on the tester. As far as browser, I think the speed for me is getting up and running. That was always the biggest hurdle and hands down Playwright wins, and they've actually made it really easy to get into pipelines. You don't even need to use the special GitHub actions that I was talking about at the Automation Guild. There's just an npx install command to install the dependencies, and you can use it on any CI runner.
[00:10:10] Joe Colantonio So let's not overlook that. Can you explain a little bit about how easy it is to install, because even with Selenium now with myself, I'm like, where's the driver? I'm on a Mac. I don't know where to put the driver. It's all craziness. Is Playwright like that?
[00:10:24] Sonja Q. Leaf Yeah, Playwright is so easy, so I can type npm install-g Playwright and now Playwright is available in my whole system and I can call the Playwright CLI but is calling Playwright, and then, I can type the word codegen. And that'll open up a browser that I can point to any arbitrary URL with a little recorder already going. And that will allow me to generate a new test code built for this Playwright test runner that I'm so excited about. It's an interactive tool like the Selenium recorder, but better, not perfect, but it's better than the Selenium recorder I remember.
[00:11:01] Joe Colantonio Right. I know they have a newer version, but the Playwright CLI, I think it just come out when you did your session. When I saw a presentation a few weeks ago, I was like, wow, I didn't even know this existed. Has it gotten better since when it first came out? Do they improve on that as well? Or is it the same?
[00:11:17] Sonja Q. Leaf They continue to improve it in ways like making it easy to set up your CI, making it easy to launch a specific browser. A lot of those features have been there since the beginning. It's been a lot of the operational make it easy to do Continuous Integration make it easy to write new tests. They have improved the explorer. So when you bring up the recorder, it used to just dump out the code in a window. But now it's actually easier to there's more to interact with. You can actually explore the DOM. You can hover over elements and that'll actually tell you potential selectors that you can type that would work for those elements just continue to improve, and their Slack channel is very active as well.
[00:11:56] Joe Colantonio I know Selenium, just came out of Selenium 4, and they have a lot of great features, Selenium is awesome. We're not dog and Selenium, but it seems like Playwright maybe does a lot more things under the covers, maybe because of the way it was designed, and because of that, you could do a lot of other things that you may not be able to do with a traditional UI functional automation script? Is that the correct assumption that I just made there?
[00:12:15] Sonja Q. Leaf That's how I look at it. It sure seemed to solve a lot of problems in one set of dependencies, and the more that it's the more complete that package has become. I'm no longer relying on an external test runner. I'm just using the Playwright test runner, and that gives me things give me really good concepts like Page Object that I don't think was in there when we talked last and that visual comparison tool to you, and that's been a big win for us.
[00:12:39] Joe Colantonio I know of Applitools so just curious to know what is, is there a difference? Is it the same? What's the visual comparison tool like?
[00:12:45] Sonja Q. Leaf It's along the lines of Applitools. You have a baseline set of screens that or objects that could be a specific element could be a whole page. Then during your test, you take a screenshot and you can compare, use a compare function. You will have a tolerance of how much you'll allow there to be differences and then you get a visual report at the end that shows you what the differences were so you can go in and review. Similar to what I've seen with Applitools and other products out there. But it's really nice to just have it in. I have one developer, that developer Hillary, they were on a project sort of on an island, and they needed to make sure that this calendar they were building, was going to generate a PDF, and they had to make sure that the PDF looked consistent every time. And we were actually able to use Playwright, generate and consume that PDF and display it, take a snapshot and use that as a baseline and then compare the next time they made changes go through tested again. Feel good about it. It's got things for us. It's been a huge win.
[00:13:45] Joe Colantonio Wow. So I was going to say, how reliable has it been? Was it really, then usually pixel? The Pixel differences and those types of tools have been a nightmare, but it sounds like it's worked for you.
[00:13:55] Sonja Q. Leaf Yep. You have to allow for some tolerance. You have to make sure that the pages that you're testing on the animations are consistent and maybe you have to have maybe here you want to have a timer to wait until an animation completes or something of that nature. If you have an image that's maybe dynamic or rotates, sometimes the same thing you can get with Applitools, you might get inconsistent overlays like over a hero image or something of that nature. But we found that it is consistent as long as we tell it to look at consistent situations.
[00:14:26] Joe Colantonio That's a good rule of thumb. So another interesting thing is we keep talking about how they're they're adding features and functionality to Playwright. We've covered CLI, which is somewhat new because it came out maybe a year ago, and they keep improving it. This visual piece is anything else, I get noticed something about an API that's able to stop and start web services for you. They have a lot of funky functionality like, does anything else that you've seen recently that you're like, wow, this is, this is, this is really helpful.
[00:14:50] Sonja Q. Leaf So that locator API was really exciting to me. The test runner allows parallelization by default, easy serialization, and you can even have all your tests in one file very easily go in one order. Even though we know that's an anti-pattern, sometimes that's the way it's got to be. The page object model has been great and they have expanded their support for more languages. So you've got good Java support, C# support, Python support, and that Python support really interests me for the future because that's where all the machine learning tools are. And with the support of a page object model and Playwright and machine learning, I think that's where my team's headed next.
[00:15:27] Joe Colantonio Oh, nice. Very cool. So but it sounds like you started off using CucumberJS because your team used TypeScript. So is that your philosophy changing now?
[00:15:37] Sonja Q. Leaf Well, I, so I learned to teach myself Selenium in Ruby because that was the most accessible language back then. And then I worked at a shop that's specialized mostly in PHP. When I joined Cloudburst, we are mostly in TypeScript and JavaScript with my gateway to TypeScript. I was able to wrap my brain around JavaScript pretty quickly and the patterns there. And then one day, the virtues of TypeScript and strongly typed languages really clicked. And even though it's a lot of set-up, sometimes it the benefits of the quality and consistency of not just our test code butter application code. There are countless having that compared to if we're writing native JavaScript.
[00:16:15] Joe Colantonio So, how about Python? If you say you're using Python, how would that work for you?
[00:16:19] Sonja Q. Leaf Well, one of my first programming classes was in Python, and I would go there only because that's where the machine learning tools are. I'm not afraid of Python, I've made my own Googlebot before that, that speaks to me, so it's not a scary concept, but I did. The main motivator for me with JavaScript was getting to better understand the domain my team was working in.
[00:16:42] Joe Colantonio You used PHP so I see how you're not afraid of Python, is PHP is a nightmare to program, for sure. Any tips for going? A lot of people think, oh, I only know Ruby. Like, like it's like they minimize something when they don't realize, you know, it's just the syntax, you know, the fundamentals that go from one language to the other. Do you find that?
[00:17:00] Sonja Q. Leaf Yeah, once you know a language well enough, you can adopt those concepts to any other language. There normally, aren't a unique con, these days in modern languages that most people are working with. There aren't too many unique constructs to that language. You can adapt what you know about Ruby. But you know, how did what a do loop does, how that might work as a closure or anonymous function call in JavaScript or PHP? It can be a moment to shift how your brain thinks, but once you learn that it's just, oh, different words that mean the same thing or different symbols that mean the same thing. It's pretty easy to adapt those skills. I even had a product where I was using the dreaded Perl for the longest period, and I ended up loving it because it taught me so much about where these programming languages came from that I don't have to deal with. Some of the crazy symbols and syntax are required to have successful PERL code.
[00:17:54] Joe Colantonio Wow, absolutely, that brings me back PERL for sure. So interesting, just go back once again to the automation session you gave in February. It was a CucumberJS using BDD behavior-driven development, so it sounds like, did you do abandon BDD? Because a lot of people always complain about BDD, it's like it's just extra overhead if you're not using it for communication. Why would be one of the reasons it sounds like you're not abandoned it, but it seems maybe like it's not the forefront, like it may have been?
[00:18:22] Sonja Q. Leaf It was one of the pieces of our process that were hard to bolt on to all the other things we're doing with coming out with three releases every sprint or so. Having BDD on top was a challenge because we weren't using it for communication. My project managers loved it, but they didn't pass it on to the client and they understood the project well enough. So then I was only doing it. It was a helpful exercise for me to understand the products as I was entering into the company. But once I was in this leadership position and I had to get mostly engineers to adapt automated browser testing, it aligned very well with some of the latest updates to elevate the behaviors in my browser test with the Playwright test runner and being able to call out the steps. API in the Test reporter gives us that same sort of information, we know it failed when I clicked book appointment.
[00:19:15] Joe Colantonio How do you know when to evolve? Sometimes people maybe get stuck in like all these are the tools that we use, we're going to stick with it. It sounds like you're a little more not open-minded, the more, you're able to adjust quicker. Maybe I'm just used to working with huge enterprises that just get stuck in the mud. But any thoughts on that?
[00:19:32] Sonja Q. Leaf You know, it is a different experience working at a nimble company. We do have a number of senior staff that has made software for a while, so when we do make changes, we do it carefully. Sometimes you just know a tool is right for the time. It solves the problem. I need to solve it right now, and it solves it in an elegant way. We wanted to make sure it wasn't doing too much just because I liked the idea of the toy or of the project or the tool. The tool actually solves the problem. So if I see a problem being solved and it makes it easier for my team to build better, more reliable code, that's an easy thing for me to sell to leadership.
[00:20:09] Joe Colantonio You seem like someone that if you actually use a different tool, a technology and it works better, you go to it. So in that type of mindset, then you know, because there's no right tool for everyone. It's like you said, it depends on what works for your technology, what you're trying to do. Who do you think Playwright is a good solution for? Is there any specific functionality, a certain technology you think it's just like, this is like perfect for this type of situation, a scenario?
[00:20:36] Sonja Q. Leaf I think it's really perfect for applications that A. Quick integration tests. B. If you have multiple user interactions, that was the place where it really excelled for me first and continues to excel for me, where I have a producer and a consumer-type user that need to interact with each other. And that has been where Playwright is, continues to shine. And it was great for me as just an automation test engineer, really easy for me to adapt, and I really liked not writing. If I've ever been frustrated by writing waits in Selenium, I'd say, Give Playwright it a try.
[00:21:14] Joe Colantonio Okay, Sonja, before we go, is there one piece of actionable advice you can give to someone to help them with their Playwright or automation testing efforts in general? And what's the best way to find or contact you?
[00:21:23] Sonja Q. Leaf The best way to find or contact me is through LinkedIn or Twitter or Instagram. Sonja Q. Leaf, or sonjaqql the as far as advice? Don't worry about testing everything. Get some tests done, that is like getting some tests done and making sure that you have some level of quality, at least to get going and to learn the API like doing creative endeavor. You have to build a number of these rather tests to really feel confident and really feel comfortable with it. So write a few. See where it starts to work for you. Then go back and flesh out those areas as you learn the APIs more.
[00:22:01] Joe Colantonio Thanks again for your automation awesomeness. For notes on everything of value, we covered in this episode, head on over to testguildcom.kinsta.cloud/a372, and while you're there make sure to click on the try it for free today. Link under the exclusive sponsor's section to learn all about SauceLabs awesome products and services. And if the show has helped you in any way, why not rate it and review it on iTunes? Reviews really help in the rankings of the show, and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed in creating end-to-end full-stack automation awesomeness. As always, test everything and keep the good. Cheers.
[00:22:46] Outro Thanks for listening to the Test Guild Automation Podcast! Head on over to testguildcom.kinsta.cloud for full show notes, amazing blog articles, and online testing conferences. Don't forget to subscribe to the Guild to continue your testing journey.
Rate and Review TestGuild
Thanks again for listening to the show. If it has helped you in any way, shape or form, please share it using the social media buttons you see on the page. Additionally, reviews for the podcast on iTunes are extremely helpful and greatly appreciated! They do matter in the rankings of the show and I read each and every one of them.
