Freeze Bugs in Time using Replay.io with Filip Hric

[00:00:04] Get ready to discover the most actionable end-to-end automation advice from some of the smartest testers on the planet. Hey, I'm Joe Colantonio, host of the Test Guild Automation Podcast, and my goal is to help you succeed with creating automation awesomeness.

[00:00:25] Joe Colantonio Hey, it's Joe, and welcome to another episode of the Test Guild Automation Podcast. Today, we'll be talking with Filip all about Replay.io. What it is? How it helps you along with Cypress. He is a Cypress expert. I can't believe Filip has never been on the podcast before. I've known him for a while, but he is an Automation Guru. He's currently a DevRel at Replay.io and he teaches testers about web development and developers, all about testing. He also is a Cypress. io Ambassador and he leads a learned Cypress.io community on Discord and has a blog where he publishes Cypress.io tips and you'll be able to find all these links in the show notes for this episode. Filip also is an international keynote speaker. I spoke with him multiple times like Applitools Future of Testing events, and he speaks all over the world as well. He's a leading expert on Test Automation in Cypress.io. And he's an author/instructor of Live Cypress Workshops. And he has taught hundreds of testers and developers about good practices and advanced concepts for testing in Cypress. So if you want to learn about some good practices and advanced concepts in testing, what the heck is Replay.io? What's this new gig he has? You don't want to miss this episode. Check it out.

[00:01:39] This episode of the TestGuild Automation Podcast is sponsored by the Test Guild. Test Guild offers amazing partnership plans that cater to your brand awareness, lead generation, and thought leadership goals to get your products and services in front of your ideal target audience. Our satisfied clients rave about the results they've seen from partnering with us from boosted event attendance to impressive ROI. Visit our website and let's talk about how Test Guild could take your brand to the next level. Head on over to TestGuild.info and let's talk.

[00:02:13] Joe Colantonio Hey, Filip. Welcome, finally to The Guild.

[00:02:19] Filip Hric Thank you. Thank you very much. And for a lovely introduction. Putting it all together. It sounds super awesome.

[00:02:27] Joe Colantonio Well, it's really impressive. And that's one of the things I want to dive into because I really like how your journey has progressed. You've gotten better and better and it seems like you're reaching higher and higher goals. The first thing I noticed is you have a new gig, Replay.io as a Developer Relations. I could be right. Though, you start more as like an automation engineer or SDET at like a Slido. And then I think you went solo. Like I said, it could be wrong, but we'll find out. And then you have this new gig, so maybe tell me a little bit about your progression and like what made you end up at Replay.io as a Developer Relations expert.

[00:02:59] Filip Hric Yeah, So you got the journey totally right. So my first full-time gig was at Slido, an Amazing Slovak Company that I think it was really amazing place for me to work and to learn because I entered there basically with like close to zero knowledge about the whole tech world. And that's where I started with testing. And I know like some people maybe not don't like hearing it much when someone says the testing is like the entry gates to testing, but it has definitely been for me. And yeah, and Slido as a company allowed me to learn and to grow, which was awesome. Kudos to them. So that's where I started. Alongside this, I started to explore Cypress. I started learning about Cypress and then one of my colleagues said, Hey, you already know enough about this that you could teach someone. And I was like, Oh, I guess I could. I went to a conference, did my first workshop, and then I gathered some courage to do the workshop on my own. And that's where we started to meet each other for the first time at the Applitools events. And I started my blog. So that's how I progressed. And at the beginning of the year, I left Slido and I went for a solo journey to do the workshop thing on my own and consultations and stuff like that. I've been on that for a certain period of time. It was fun. It was a lot of hassle. It was kind of crazy that I did my own marketing, I did my own home page, I created my videos. I did quite a lot. There's a lot if you want to go solo but it is, it was really, really rewarding. And yeah, but a little time before that I met Andrew Knight, who was a DevRel at Applitools. Shout out to Andy. A great friend of mine. He visited last year for-well, he was close by, so I invited him to stay over and it was a right in this very room when we talked about what is it that I actually do, where am I progressing? And we found out that we created together that tagline of mine teaching developers about testing and testers about web development. And I have been feeling that this is the journey I am on and this is my mission because I never knew which bucket I fall in, whether I'm more of a tester or more of a developer. Somewhere in between. I don't feel like a developer, but I also maybe not always feel like a tester. So that is my sweet spot. And yeah, Replay.io was looking for someone like me who's doing exactly that. So that brings us to the present day.

[00:05:51] Joe Colantonio I love it. And I love like the way I grow as I watch people that are ahead of me and I see what they're doing and I try to emulate it. I think if people listen to this, you're a great person to emulate because you start off like you said, you didn't really know testing, you learn testing. And I don't know if you suffer from imposter syndrome and things like that, but it seems like you get over whatever hump may have been like, I'm not a Tester, I'm not a developer, but you taught yourself and then you start to create content around a topic. And then I probably, I could be wrong I assume Replay.io found you because of all the content you were creating. Is that true?

[00:06:24] Filip Hric Yeah, probably. They reached out to me because they have started this new product or this new area of where Replay can be used and that's test suites. So they were looking for Cypress expertise, and Playwright expertise and tried to get feedback on whether the tool they are developing is useful for someone like me as a Tester. That kind of brought more and more conversations after that.

[00:06:52] Joe Colantonio Yeah. It's not for you going on your own and doing it or sharing your wisdom. I mean, probably you wouldn't have had that connection. I think especially in this economy, what you did is great and people should do more of.

[00:07:02] Filip Hric I'm sorry to jump into that but there is a little thing that I would like to maybe say now that we are public and people might be listening.

[00:07:12] Joe Colantonio Sure.

[00:07:14] Filip Hric Because as we started like put all of those successes together, it sounded superhuman-like. And the journey that I went through, I tried to live by one principle in terms of like content creation is that I always wrote a blog about something that I was currently learning, it didn't have to be like long-form, super amazing stuff. It was just this tiny thing that I learned, whether it was that day or that week because I knew that if I found a blog post like this one two weeks ago, I would have learned that two weeks ago not today. To anyone out there who's considering writing a blog whether on Medium or on their own, I really encourage you just take a thing you didn't know two weeks ago and write about that. It's going to help someone because you're on a journey. Some people are in front of you, some people are behind you and you're going to help those who need the information that you know.

[00:08:17] Joe Colantonio Absolutely. And that's actually how I started. I started a blog JoeColantonio.com talking about WinRunner, LoadRunner, and QTP. There were no blogs back then on that stuff and then just snowballed to what I'm doing now. Great advice. Not only you helping yourself because as you teach, you learn yourself, but you also it's almost like a little breadcrumb you're throwing out that you never know what it's going to lead to. For me, it always led to something better and better along the way.

[00:08:41] Filip Hric Yeah, exactly.

[00:08:43] Joe Colantonio All right. So I didn't know Andy Knight was also a therapist. I need to talk to him next time to help tone in my messaging. That's awesome. Andy was doing a DevRel position, I guess at that time he's no longer doing it. But does that help? Opened your eyes to say, hey, maybe I want to be a Developer Relations expert as well? Or was it just coincidence that all of a sudden you had another company looking for a role like Andy had?

[00:09:07] Filip Hric I actually knew for a while that I would love to land a DevRel job because, I mean, a couple of years ago I didn't even know what DevRel is and that it is a thing. Well, DevRel is sort of the area is very loosely defined. But what I did and what I enjoyed doing was creating content about Cypress, sharing videos, talking at conferences, talking to people about their experiences, and helping them troubleshoot. This was part of what I really enjoyed, and I didn't know that this could be a job. And DevRel, at least, part of that DevRel job description is doing this. I felt like, oh, if anyone would hire me for DevRel, well not anyone, but if there was an opportunity, I would gladly take it because it seems like something that I could maybe be good at. But as I mentioned, not anyone. Actually, I would like to move in an area that I really enjoy or maybe talk about a product that I really enjoy. That's what I've been doing so far and that's why Replay.io was a very natural transition for me because I was already creating content, going to conferences, and doing stuff like that, and Replay sort of fell into that space where I could, Oh, this is a really cool product. I will be able to talk about it and not lie about anything I'd say it's super, super awesome, and actually is, so yeah.

[00:10:43] Joe Colantonio No. So like you mentioned, maybe a lot of people don't even know what a DevRel is. I know it's a position that's been on for a few years now. I mean, it's been around forever, but it seems like it's growing in popularity. What is a DevRel for someone's like, Oh, wait a minute, this might be something I like as well?

[00:10:57] Filip Hric Well, for the past couple of weeks, I've probably given like 20 different definitions like to my parents and to my wife and friends. I tried to do one right now and it is probably going to be different from those 20 that came in before. But the way I understand DevRel is someone who's at DevRel position, is a public figure of that company, is trying to do a little evangelism about the product that the company creates, which may sound like marketing, but as we know, marketing doesn't really work with developers, right? You cannot just put out an ad and hope someone clicks on it. And the answer basically is in the name of the position Developer Relations. So what I want to do is to create good relationships with testers and developers, go to conferences, and provide useful content that will not always be about Replay, but maybe about a broader topic like end-to-end testing, end-to-end testing flakiness, Cypress, Playwright, other frameworks as well. That's probably basically going over the definition because I'm just defining it by explaining whatever thing I would do, but that's kind of how I did that for my friends and for my parents, etc.

[00:12:19] Joe Colantonio Nice. So is it meeting your expectations and what you thought it was and then now what you are doing, It's in line of what you were hoping it would be?

[00:12:27] Filip Hric Yeah. I've been there for weeks. I think this is my 5th week to time of recording so far. It's great. I work with a team of amazing world-class engineers. What's talked about imposter syndrome, right? They're just amazing, super smart people. I love talking to them. Currently, I'm working on new documentation, so I need to think very deeply about what steps people need to take in order to get a good understanding of what Replay.io does. And that's been always sort of at the center of when I was writing a blog, when I was preparing a talk for a conference. So far it is I'm at my sweet spot.

[00:13:12] Joe Colantonio Awesome. I guess the next logical question is what is Replay.io? Is it a newer tool? Because I just started hearing about it from you, watching your tweets and your blogs and things about it.

[00:13:21] Filip Hric Yeah, that is something that I also need to learn how to convey, like how to put that message forward, because Replayi.io is a tool that is sort of in a category of its own. So if I were to go with maybe let's do this with a couple of basic steps, right? So the most basic definition of Replay would be that it's a browser, right? But it is a browser with special capabilities. So normally when you run a browser, you open a page, you browse everything, you wouldn't use Replay like this, you would use it for your development or your testing workflow because Replay the browser has the extra capability. It has the capability to record your application or record the web page or the application that you are visiting. And so the recording that it creates is, first of all, it's a video that's fairly understandable. Second of all, it's something like a trace viewer in Playwright or the timeline or the test Replay that Cypress has. And on top of these two, you actually get the application runtime. Not only it record DOM snapshots and network calls, and calls the logs, etc., but it actually follows the flow of the information of the application. Let's imagine I'll give you an example. Let's imagine you have an application and you're debugging that application. Not sure what's happening, something's wrong. So you go to your VScode or some other editor and you add a console.log to understand what's going on here? Try to see like, how is my application actually processing that data? With Replay what you can do? When you create that recording of your application, you can actually add the console.log after. So you first record everything, Replay is going to capture everything and then, you will go to your code and add the console.log after. But you don't do that in VScode. You do that obviously in the Replay browser. So to sum up, basically the experience with Replay will have two parts. First one is creating that recording and second part is like all of the dev tools that you might have in Chrome or Firefox and then have those tools like interact with the code that's been recorded.

[00:15:52] Joe Colantonio Okay.

[00:15:53] Filip Hric Does that make sense?

[00:15:55] Joe Colantonio It does. How does that help though? If someone is using it? It sounds almost like telemetry of observability to me as a layman here where it's able to I guess if there are issues, it's able to bubble it up because it has that tie into the application. So rather than having all these things, you look at it, it's like kind of, I would assume condenses it all into one location?

[00:16:16] Filip Hric Yeah. The way that you might use a Replay there were actually two big use cases for us right now. I mean, there are more, but the two are, like, most prominent. The first one, imagine you are a tester and you want to replicate a bug, right? If you try to replicate something and then you set out those steps, send them to the developer. And the developer says, Oh, sorry, I can't replicate this. It probably doesn't exist. You imagine the bug. And the tester gets frustrated. Well, I gave you the exact steps. What the hell is happening? It did happen to me. Well, if you were to find those steps using Replay. So if you open instead of Chrome, you opened Replay browser, you would record that bug. And if you were to send that recording to your developer, they will not only have like the video and DOM snapshots of what was happening, but they will actually have the code. So the source code off that recording and in addition to seeing that source code, they could add a print statement or a console.log to see. Alright, so here's something that didn't match up, right? There was some timing issue. Something happened here. I guess I'll add a print statement to see what the information was like, what this function return, or what went into this function. And this can reveal like some timing issues and other kinds of problems that applications today have. Another use case for that, since you have that concept of having like these robust recordings that you can work with and as a great example of how it can be used is that since Replay is a browser, you can use it for your end-to-end tests, so you can hook it up to Playwright or Cypress or WebDriverIO and use it as the browser of your choice. And those recordings would get recorded manually or not manually, automatically. And when you have a flaky test, you have one test failed, one test passed. It's the same test. You can do a comparison and see what went wrong and not only see what went wrong but actually examine what was happening in your application. If I were to put it maybe back up a little bit on how to understand this all is that when we have like a front, when we have an application, we have usually like the front end code, which is like React, some kind of JavaScript, CSS, whatnot. Then we have the backend. So the API, which we are communicating with, and then we have the data layer. So your application is just like a bunch of functions. They will take the data maybe from that backend and then process it through the application. With Replay, you can see that data and how it is processed. And to understand like how your application actually worked with the data that was flowing through. So if you have a situation where you have a function A and a function B, and function B actually uses information from a function A, But in some cases, you might get into a situation where they don't trigger in the right order. You might be able to find that problem with Replay quite easily because you'll console.log out information from function A, and console.log out information from functions B and C. Oh, okay. There was a timing issue and we actually have a flaky app.

[00:19:54] Joe Colantonio Back in the day, I was a performance engineer and I had a tool called LoadRunner and had a plug-in. I forgot the name of the plug-in where if you ran a test for performance test and you saw a degradation of performance spiked up, it allowed you to drill down and can even tell you like the sql statement when you ran this test, this is the SQL statement that took X amount of time to run so you could tell, or rather than just guess because it has it's able to read the code. It was able to point to the exact SQL statement that was causing the slowdown. Sounds something like that, is that sound correct? Kind of what it's doing?

[00:20:27] Filip Hric I think there might be some similarities.

[00:20:30] Joe Colantonio All right, cool. Cool. So how much time does this save someone then? If someone's listening to? Like, have you seen or heard? Like, Wow! Without this tool, I would never have been able to find this flakiness because flaky is notoriously hard to debug.

[00:20:45] Filip Hric Yeah, I don't have the numbers yet, but recently I made a poll on blogging and asked testers. Do you spend a bigger portion of your day debugging your tests or writing tests? And the responses said that 61% actually spend more like larger part they're debugging. The potential is definitely there. And we've been working with some clients or potential clients right now and they are running their Cypress tests. They definitely have some flakiness, as you would with any tool, and we've been able to drill down like without even knowing too much about their code. But we were able to drill down different sources of flakiness like there was an AB marketing campaign going on and that threw off the tests. There were some false positives that actually resulted in flaky tests. There were some timing issues. Some model windows popped up too soon and all kinds of different sources of flakiness. And we could see those happening inside the code and we could do that in just like a couple of minutes because we could easily narrow down like the moment where we need to look and also the functions and the components that are inside the application to see what they were doing, how they actually processed the data, what went in, what went out. And we were able to pinpoint here exactly is the problem. This is what needs to be fixed. And other this sort of reminds me of another poll I did recently because I asked a very theoretical question. If say, you have non flaky app and non flaky server, what do you think that would do to your tests? Make them less flaky, more flaky, or the same as before? Now, 79% would tell it would be less flaky, which is interesting because we are talking a lot about flaky tests, but we don't really talk too much about flaky app. And yet the result of the poll sort of indicates that we should probably be taking the source code to the application code into the equation. And that's what Replay.io does because when you have a trace viewer in Playwright or the timeline in Cypress, you'll get insight into what the test was doing. But don't really get into too much insight into what the code of the application was doing and you get that with Replay.

[00:23:15] Joe Colantonio How is this to be configured then? like what do you need in order for this to work? Is it just the browser? I assume you have to have some sort of sniffer or a tie into the code itself.

[00:23:23] Filip Hric Well, the integration is pretty simple. Like if you know Cypress, you run it with npx Cypress run, and then that would choose the default electron browser, if I could call it that. If you were to do npx Cypress run -- browser and then choose Replay chromium, which is like the chromium browser with Replay capabilities, then that would be basically it in principle if I were to dive into the code there say like a plugin installation you need add something in to support file, then also to your plugins file. But the basic principle is like yeah, just use this browser and you'll get the recording.

[00:24:05] Joe Colantonio So I'm just assuming because it's a browser, if someone's doing like if you had a bunch of testers and we want to just explore a particular part of the application and test it really well, I want to do some exploratory testing sessions. Would you be able to use Replay.io to help you with that as well as you need to use an automated tool?

[00:24:22] Filip Hric Yeah. You will be able to use it as a standalone browser. What you can do simply is to open this Replay browser, click on start recording, and click through your application. You'll reproduce your bug, for example, it'll stop the recording, and then what you can do is to send that recording over to your developer. They will open that recording and along the video and the comments you might add into that video, they will have the full source code as it was executed during your session. There will be no more like, Oh, it doesn't happen on my machine because now we have the log of everything that happened on your machine, not only the actions and clicks and types but the actual source code, which can now we see and we can take a look into every function, what it has returned, what information was passed into it. Basically, developers at that point will have everything they need in order to fix that issue.

[00:25:27] Joe Colantonio I'm looking at my notes and it sounds like this we already explained it's almost like little time machines are being used and in your end-to-end testing. Cypress timeline, Playwright Trace viewer, all this, it's like a point in time that you can go back and see what happened, have all the logs for to help debug what happened during that that point.

[00:25:44] Filip Hric Yeah exactly. But in addition to what you might call a session Replay which is what Cypress and Playwright does in addition to the session Replay that Cypress and Playwright has, you will have to source code as well and also like the source code execution. So yeah every function what it has returned, what it has been passed into, you can look into that.

[00:26:07] Joe Colantonio All right. Another dumb question, it's a browser, so you can use it for exploratory testing? You mentioned Cypress and Playwright. I assume then you can use it for any automation tool that can invoke a browser then? That can invoke Replay.io. Right?

[00:26:19] Filip Hric Yes, exactly. We also have support for WebDriverIO. There is experimental support for Jest. I'm not 100% sure about the implementation of Selenium. I would need to take a look into that. But I think in theory, yeah, since it's a browser, you could use it. But with tools like Cypress and Playwright, we actually have like an additional UI that you can use. You don't only see the browser and the execution and the video from the test run, but you actually have a side panel and you can go through each and every command and jump to the point of the code, jump back into the point where, for example, the click triggers or to the point where the typing started or something happened, some API call was made, etc.

[00:27:13] Joe Colantonio Cool. I also assume this would help out of the .. division between testers and developers. Seem like roles are kind of morphing together, but I assume if there is a team of dedicated testers and a team of dedicated developers, it's almost help the communication, which tends to be the biggest hang-up I've seen with a lot of teams. That's something you see as well.

[00:27:31] Filip Hric Yeah. Definitely. Because if you are a tester, if you want to file bug reports now you're basically relying on the execution steps, right? At least that was the experience that I had, right? If I had good execution steps and my developer friend could replicate the bug executing those same steps, that would be great. But oftentimes it would happen that if they would go through the same path, they would not be able to find the same bug. And this way with Replay, you can record that and have the exact same steps and the exact same thing available to them. I think in communication is going to help quite a lot in teams. Yeah, I definitely agree with that.

[00:28:20] Joe Colantonio Nice. All right. We're heading into almost 2024. We're in October when this will be released. Cypress versus Playwright in 2024. Who do you think wins? Is there not even a battle or are any other tools you see potential for in the future that I think people need to know about?

[00:28:36] Filip Hric I think it's definitely a battle, right? Everyone wants to take a bigger piece of the cake. My prediction, honestly, is Playwright is going to continue on growing and I feel that maybe the year 2024 will be sort of the year where we could be able to tell the future of what the future of those two frameworks will be. I think if the Playwright adoption is going to grow as fast or faster than it has last couple of years, then Cypress obviously needs to respond with something amazing. And if they will have a good response, I think it still will be an interesting race. If they don't, it might in a couple of months or a year, it might be the case that Playwright will have more users. Right now, I think Playwright has like a one fourth or one fifth. I don't remember the numbers exactly of what Cypress has. As we know, the industry moves a little bit slower with the adoption of new tools. I think next year will show what the upcoming year will be. If Playwright continues their rocket growth, which they seem they might be. I think Cypress will be in for a huge competition.

[00:29:59] Joe Colantonio Wow! Hot take for someone who loves Cypress. All right. Very honest.

[00:30:03] Filip Hric I love Cypress like nothing has changed on that part. Honestly, I think it's a better tool. I enjoy working with that quite a lot. I've been using Playwright for a couple of months already and I've been slow, learning slow, so that's why I prefer Cypress. And yeah, that's something to consider like how fast are you able to debug and how fast are you able to write your test? Because we know that Playwright is faster than Cypress. Objectively just executes faster. But the execution part, that's actually the part of the day we don't do. We just hit the script and it runs for ourselves. And yet as a community, we're quite fixated on that. I think the very important part is being able to write your tests fast and being able to debug them fast because they will be flaky like every time. When we were talking about end-to-end testing, we will have that problem in the future. The sooner we can fix those tests and find out what was the problem, the faster we will move and the better coverage we will have, the better trust in the tests that we write, we will have, etc..

[00:31:20] Joe Colantonio Wow! Awesome tip. I definitely agree with you. Okay, Filip, before we go, is there one piece of actual advice you can give to someone to help them with any of their automation testing efforts? And what's the best way to find contact you or learn more about Replay.io?

[00:31:37] Filip Hric This podcast is called Test Guild, right? So testers are listening and my number one advice to all the testers is go write an app, try it for yourself. I think that experience is really eye-opening. It's going to help you understand what's actually going on during the development. It will build maybe a bigger empathy towards our developer friends, but also and I think that's most important, it is going to give you quite a good context and it's going to make you a better tester because the decisions you make as a tester on what to automate, what to maybe skip if those aren't informed by the knowledge of the code and how your application works, they are going to be more effective. So number one tip build an app, try it out for yourself.

[00:32:30] Joe Colantonio Love it. And the best way to find a contact you, Filip? if people hear this and like, Oh my Gosh! I need to follow this guy or I need to reach out about or learn about Replay.io?

[00:32:38] Filip Hric Yeah, I'm on LinkedIn, I'm on Twitter. That's where I'm mostly active. Or you can also go and check out my home page Filiphric.com. That's where you find me. And if you're interested in learning Cypress, then you can go to my discord which you will find link on my page. Filiphric.com/discord. And if you want to learn about Replay then go to Replay.io. We also have a Discord channel. So if you're eager to try it out or you stumble upon some problem and need help, I'm hanging around in that discord so feel free to reach out.

[00:33:15] Thanks again for your automation awesomeness. The links of everything we value we covered in this episode. Head in over to testguild.com/a473. And if the show has helped you in any way, why not rate it and review it in iTunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe, my mission is to help you succeed with creating end-to-end, full-stack automation awesomeness. As always, test everything and keep the good. Cheers.

[00:33:51] Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps, and automation professionals in the world, we'd love to have you join the FAM at Testguild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real-world value that can improve the skills or solve a problem for the Guild community. I love to hear from you head on over to testguild.info And let's make it happen.