WebLoad vs LoadRunner vs jMeter (Who Wins?) with Paul Kanaris

[00:00:01] Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability for some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain Podcast and my goal is to help you create DevOps toolchain awesomeness.

[00:00:19] Are you struggling with JMeter's limitations? And Load Runners overwhelming complexity and price? Wonder if there's a better way? Well, today we're going to break down how Web Load strikes the perfect balance between ease of use and robust performance testing at an affordable cost. Don't miss out on transforming your testing strategy. In this episode, performance testing guru Paul Kanaris, with over two decades of experience, shares insights on selecting the right performance testing tool and ways optimize your performance engineering efforts. And if you're looking to move away from JMeter or Load Runner, definitely try it for yourself. Head on over to testguild.me/loadtrial and follow along and see how it can help you elevate your performance testing efforts. You don't want to miss it. Check it out.

[00:01:03] Joe Colantonio Hey, Paul, Welcome to The Guild.

[00:01:07] Paul Kanaris Nice to meet you, Joe.

[00:01:08] Joe Colantonio Great to have you. I guess before we get into maybe a little background, I know you have a lot of experience, but for the folks that don't know a little bit more about yourself.

[00:01:15] Paul Kanaris I've been involved in software testing for over 20 years, claimed that I can do all things testing. I've done functional testing manually and automated, started out with the Mercury Suite of tools, Implemented Test Director, Quality Center, LLM, Load Runner 6.0 so that'll kind of give us a reference of how long it's been. Worked a lot with Load Runner over the years. As we know, Load Runners like the 10,000 pound gorilla that does everything performance testing, but it's still the gold standard. That's where part of that comes from. I've been an instructor for performance testing as well as the testing tools. I am ISTQB certified the foundational level as well as the test manager. And my current role, I actually lead a global QA team of about 73 people spread around the world. We've got a small contingent for performance testing right now that we're hoping to grow over the next few years.

[00:02:17] Joe Colantonio Nice. So I guess, you mentioned Load Runner and I start off a Load Runner myself as well. But now I've been investigating more and more of Web Load. Just curious to get your idea of know I believe in your current position. Have you made a switch from Load Runner to Web Load and if so, why?

[00:02:35] Paul Kanaris Actually went from what the company was using was JMeter. I looked at Load Runner, but of course Load Runner right said it's still a very pricey set of tools which makes it very difficult to get a company to buy into. But JMeter, of course, is not a tool. If you're looking at end user experience, many people go but JMeters out there, everybody uses it. I always say use the right tool for the right job. If I want to know how this is going to work with thousands of users, they're not interested in how fast the API works. They want to know when can I do my job? That was one of the things that we did. And basically, when I looked at Web Load here is I brought up the Load Runner documentation as a reference to say if it doesn't do these things, it's probably not a tool to use. I looked at something that people think is a very good tool and that was something that I encourage people. Then you take that, you bring that in, and then you start going through your checklist of what will this tool do? Doesn't meet that. Then it starts becoming down to if you can get a tool that it's 80%, 90% of what's there. I also consider things like how extensible is it? If it's not extensible, how good is the company at what I might need to have at it or not? And that's one of the things that I've learned really starts helping you understand what you can do. That's kind of what I did when I came on board. Now, of course, we looked at the cost and you need to do a good ROI and you need to start looking at though even if you can buy a very good tool, it may not be as easy to implement or as fast as you want. So don't over promise.

[00:04:25] Joe Colantonio I think you're right about it. A lot of times people say, well, it's free. But from my experience, using something like Web Load is full functional where JMeter is very, it's not enterprise wide that I'm aware of that can handle what I'm used to with a tool like Web Load.

[00:04:39] Paul Kanaris Right? JMeter is very good at doing checks on very small activities. It's when you start getting into the more complex like were a supply chain management system, you're starting to talk about some very big amounts of data that need to be moved around. And again, I go back to what am I trying to do? JMeter was designed literally to say, are my back end calls working? It was not designed to say, how long does that presentation layer take? How much time is that consuming? Where am I really seeing the impact so that I don't get any of those nice phone calls at 2:00 in the morning from my CTO screaming that the system is going so slow that he's got the CEOs all screaming at him. This is what you have to really start looking at is understanding what the tools do, what your goals are versus their limitations. When you go out there, you'll hear, well, you can embed Selenium. Well, that's fine. It's still not check in the return and the responses of thousands of users. Whereas with Web Load, you've got virtual user, these are real virtual users. That's why you can only add about 150 per generator because they start consuming resources. Whereas with JMeter, you can spend up to 2,000 users on one box that doesn't use that much memory. That would be a big savings people say. Well, okay, fine, what am I going to get from it? And that's where I keep stepping back. The other part is how much do I need to do in JMeter to start extending to connect to things like we're going to be keeping cloud watch in from AWS because everything we do is up in the cloud. Well, if I've got to do that in JMeter either need to rely on somebody building that as a plug in or figure out how to do it myself. When I get something like Web Load or other third party tools that are more industry-based, I can get that already and therefore, I save a lot of money and time where I'm not trying to figure it out. Performance engineering shouldn't be focused on building tools, and I call it performance engineering, not testing. People don't understand the difference, but one is running scripts to run scripts. Engineering is developing that test, developing the methodologies and understanding what you're getting out of it so that you can come back and say, Based on what I've executed, here are my potential bottlenecks that I see, and here's where I think we need to start looking. And also adjusting those tests. How do I find out if the background servers are chewing up my system or not? Once I do that, what do all these metrics mean? That's the engineering aspect. If you can explain what you're really coming out with, you're not going to get anywhere. And that can be really hard with something like JMeter, and that's where Load Runner had a lot of advantages and that's where Web Load steps up to the plate, I get a lot of data. I can look at it, I can figure out where I can be, I can build new graphs, and I can do some things like overlaying my transactions against my server response times. I can go ahead and see those spikes and start looking at what's my relationship, and I'm doing this real time. One of the problems with JMeter is, yes, you can bring it together and you can try to pull it all together, but you've got to actually get that data from the server, bring it in and hope the timestamps are right. Whereas, a third party tool, if it's designed correctly, it should bring that data in to the data at the same time. Now when I'm looking at that spike at X milliseconds of the run, I should be able to go check my resources and say, where is my problem? When you've got a large system like the company I have and most companies that will benefit from something like Web Load, I've got about 45 servers. I've got to keep track of, each doing different things. I've got my application layers, I've got my database, I've got Elasticsearch servers. All these things are interacting. Where's your bottleneck? Well, if you can't precisely gather that, and you can imagine bringing in 49 servers worth of metrics. If you had to pull those from each server, bring them in. It would be extremely challenging. Whereas Web Load, it's all pulled down, I can look at what I want, I can remove variables out of there, I can remove measurements, and then I can really look at start focusing what do I think is my problem and present those findings clearly to my development team.

[00:09:33] Joe Colantonio I love that. And like you said, a lot of times ROI people hate hearing that term, but a lot of times you'll see a free tool and they think it's free. But I think you laid on a lot of good points why a free tool like JMeter isn't really going to save you money in the long run at an enterprise level is what I'm hearing.

[00:09:48] Paul Kanaris Most tools that I see that are free, people have to reflect on and see it's not free. There's a lot of overhead. If you want to do something with performance engineering, you have to build a lot of infrastructure around JMeter in order to get it to work within your environment. Now, if you're working with some strange environments and I'm working with AWS, I could be working with Azure, Web Load supports both. I can go ahead and work with the cloud. It's now got some artificial intelligence, which I've got to work with them a little bit on. They decided to go with some of the open frameworks out there, going to have to work with them on a little bit on how we get with a restricted cloud, because that is one of the things that you've got to be looking at, two, with the tools is what if I have some regulations such as GDP and a lot of other restrictions? How do I use a tool that's open source? How do I get them the information that I need and even the tools that I do pick? How do I make sure that it's compliance? That's going to be another layer that you've got to add on top of that. And how do you keep all those libraries in sync. One of the things I like is if I don't have to do all this heavy lifting of building some extra extensions, then I don't have to go ahead and maintain all the security potential risk because all of our stuff is actually scanned. Make sure that we're not leaving anything out to the internet. If I was actually doing that all, I'd have to have them look at all my libraries, all the stuff I'm compiling. Whereas, what we get from someone Web Load, we've already got a security certificate. They don't need to go ahead and start digging into the tool to figure out if I have some vulnerabilities. And I'll say that because I have an open source framework, something that I was kind of forced to build. I would prefer something different. And it has cost us a lot of money and a lot of heartache keeping track of those libraries, keeping track to a build in all that code, building all the integrations. Think about that when you're dealing with performance engineering. The more things that you have to build it, complete your job. The less time you have to do the job. Therefore, is the ROI really saving you money? And I have to look at a tool like Web Load it. It was easy for me to do it. It cost a lot less for me to have Web Load on my annual purchase than for me to spend my salary, plus some others to build all the stuff I would need to with JMeter.

[00:12:26] Joe Colantonio All right. So in this round, I'd say Web Load versus an open source solution. This round goes to Web Load. Round two, I'm going to call Web Load versus, say, Load Runner. We talked a lot about Load Runner already. So you came into an environment that had JMeter and I assume you start exploring Load Runner and other solutions and you've used Load Runner before. And I'll be honest, I haven't used Load Runner Hardcore since 2006, I don't know if it's changed at all. Is there any functionality or pain points that you had in Load Runner that when you looked at something like Web Load, you said, wow! Web Load actually takes care of these pain points as well.

[00:13:00] Paul Kanaris Some of the things that I looked at was as an example, Web Load, you can bring up their dashboard in the browser and you can actually open that up to where others can be looking at it. You can only run it from the original console, but others can be looking at it. Well, if you want that Load Runner and I looked at Load Runner and I actually used Load Runner probably about 6 or 7 years ago. You had to buy that. It was an add on. You had to buy different. What type of thing are you testing? You got to buy each of the pieces now, of course, you could buy the big bulk package. It gives you all the different protocols, but then you spent a lot of money. That's a problem. When I start looking at correlations, are they easier and Web Loads and Load Runner? I'd have to probably say that's why I like what Web Load does. A lot of the features are comparable. When I start looking at features, the majority of things that I did in web bloat, they worked pretty good. But of course, I didn't dig into everything that Web Load could do because I didn't need it, why worry about it? Same thing with Web Load. I'm sure there's a lot of things I don't need for the current testing project, but those things that you need from a user perspective for performance engineering. In general, I can say you're going to get complementary features. I do think that there's a much cleaner look to Web Load as far as the browser piece. As far as the console, I can tell you the console in Load Runner, they spent a lot of money building that because originally that was the only way you could see things. I would have to say the console was in some ways a little bit nicer. I could bring up more metrics on it. I could make it look like I was basically being an air traffic controller with all the things I could bring up. That doesn't quite happen in Web Load but do I need it? And I think that is what you got to look at is there nice things, but are there things I had to have or was it really more along the lines? It helped me while it was live to try to do some preliminary analysis because that's what I needed to do for that job. I would say you've got a pretty consistent 1 to 1 at least Web Load does the same thing is Load Runner. And then I think the browser piece really starts opening up the doors where you can start sharing and building collaboration because the more you can build that collaboration with an architect, especially with my world, it's global. I need to have my architect be able to look at that. In matter of fact, my architect, I don't think even lives in the same city I do. This is really becoming the world that we need to have those tools built in with collaboration. And we shouldn't have to be buying extra plug ins for collaboration. I think that's one that really kind of sets Web Load aside.

[00:16:02] Joe Colantonio All right. So I'm having flashbacks because every time I needed to do something at my job, I used to go, it's not available. I need to pay to unlock this or it was crazy. And it doesn't sound like Web Load has that, which sounds great because you can get your job done without saying, by the way, we need more budget to buy this other protocol or other add on.

[00:16:22] Paul Kanaris I haven't run into that. And it's just like if I had looked at Web Load of my previous company, they just decided not to. It was why I was already familiar with Web Load when I came to this position. They built things for me that I needed at the previous company and we've had to make some adjustments and I can work with their support people and I even actually work with their developer, one of their major developers. Just most of the stuff I get into. It's not that easy because I've been doing it long enough. I can figure most of the ways around things. When I get stumped, it usually takes some help. They're there. They're not asking. They're not holding their hand out for more money. They're not saying that's going to be special support. You got to pay X dollars. I had that with I should say with Load Runner where they would say, well, we can probably do that for you if you're willing to pay us. That's not what I experience with Web Load. Now, I'm going to ask for, like I say, the private AI stuff. They may have to hold off on that because they may not be ready for it or they may say no. And that's okay. But I believe if you can justify it with them, they'll do things. And that's why you don't want to go ahead, though, and say, I want to work with this special, unique little A.I. tool. It probably will tell you know, for sure what I'm looking at is Azure. That's probably something that they would go with. The only reason why I'm doing that is I've talked with my company and with my automation and my test management tool. They want to implement A.I.. I've got to use Azure. I'm like, Well, if that answer is my A.I., then I'm going to try to do the same with Web Load and I'll talk to them. So I think that's one of the big things, Joe, that you bring out really shows what you want with the tool. Are you paying for every little thing or are you paying to get a tool that does what you need? But also, I think the other thing that you got to look at is what is the support like? Can you get help? I remember the Load Runner a lot of times. I could wait a couple weeks before I got a hold of people. And when I first used Load Runner, I was working for a major health company, the largest in the United States. And I had to wait. I hear you there.

[00:18:39] Joe Colantonio And I think that's one of the stellar features of Web Load and Rad View from what I hear is there help is top notch. And like you said, they don't nickel and dime you. They're willing to work with you right away, which in my experience has been when you're doing performance testing, like you said, you never know who you're going to run into. And having that type of helpdesk or that help support is a big deal.

[00:18:59] Paul Kanaris And we've got not just one product. I've actually got about 14 different products that we're going to be bringing over, and these are all from different companies that were consumed or purchased. You can probably understand I've got some that are on DB2, some that are on Oracle. We've got some on my SQL. We got every flavor of databases as a back end, every flavor of application server. We've also got some very unique, as I'm sure you're used to, Joe. Developers can be rather creative in how they build things. Having the ability to have a tool that is flexible, but also you're going to run into something. It could be the way that they're returning information. You're trying to get the data out so you can feed the next screen, or maybe you're trying to do some changing and you just can't find it. I've had that problem and I can submit a ticket and they will get someone online with me if necessary. And that's where I've worked with their nature developer because we found out it wasn't that easy. Well, he helped me and within probably 10 minutes he was like, yeah, just do this and this and I'm going to go on. Who would think of that? And that's the kind of help you want.

[00:20:14] Joe Colantonio And I think it also outlines an enterprise need for an enterprise wide solution. I'm used to working at health care companies, banks, financial insurance. And you never know what you're dealing, what protocol is going to pop up, what back then you're dealing with what, like you said, what the developers coded up. It is a big help to have that help there to be able to get through those type of issues for sure.

[00:20:36] Paul Kanaris Yes, that's probably the biggest thing when you're at the enterprise level. When you're with a small company, you usually can adjust your application to work with the tool. Instead, what you want is a tool that will adapt to your environment. And when you've got an environment that eventually will probably be built in a consolidated system. We're going to be looking at more like about 150, 200 systems all talking together. And that will be a very massive load. We've got a lot of unique constraints, which you have in the enterprise. We've got GDP, so we've got those regulations, we have the regulations here in the States, and we have to be very careful because we also our supply chain management support some more unique companies that deliver things to the government that have security clearances. We've got the whole gambit of all the challenges you can have from a customer base. But remember, those customers are the ones that are going to drive what you need to do. Each of those opens another potential risk that that tool will not do what you need. The best mitigation I can tell you is have a tool that is either adaptable for it or a company that is willing to help adapt their tool and or adapt at all.

[00:22:02] Joe Colantonio I totally agree, I hate using a tool and you have to go to their what they recommend, their format or their flow, it really will mess things up if you have to conform to the way they do things rather than it performance the way you do things. I love this.

[00:22:16] Paul Kanaris And I found Web Load does plenty of that. You can use it however you need. We've got some very unique ways that our current the main product we're working on, which is one of our biggest the way it pulls data back. Every company is unique. Even though you're searching for, in our case, a product, every company brings back a different way, a different payload, everything. This is all the things that you're going to want to think about when you're looking at what tool do I want? But I do strongly suggest that if you are looking at tools, go with the one that you know is the best and say, okay, so what do they offer? And then start looking around and say, what tools can I get that will offer that. And start cutting down your list.

[00:23:05] Joe Colantonio Another thing you mentioned once again, you're in enterprise. You never know what you came back. What you developers are doing is the big thing of thing about correlation, I know is when I was a performance engineer, correlation was always something that kind of could struggle with depending on what you were testing. And from what I hear from Web Load is the correlation engine is really superb. Do you have any experience between Web Load's correlation compared to Load Runner and which one you prefer?

[00:23:28] Paul Kanaris I would actually think the Web Load correlation was a lot easier once you understood how it was set up. Coming from Load Runner, I have one preconceived notion about how it was set up and where I learned it from though of course was health care. So in health care, we had to do a lot of things with using that correlation to get information out of the headers and the streams of data that really made it unique. Now, most of the time when I worked with other products after that, it was pretty easy, pretty straightforward. Coming now back to a supply chain management where we've got a lot of data, we've got a lot of customization. The correlation again has become a little more complex. When I looked at it, Web Load is easier. Now, there were some things that I needed some help with, but there again, that's where it's great if you follow everything they tell you and you still can't get somewhere. I could make a phone call. I can go ahead and send an email my go through the support desk, then they get someone on board. But it isn't two weeks from now. It's usually either the next day they'll send me a snippet to try it. If that doesn't work, usually we go through a couple tries with that and then we get an appointment set up. It were real time there on my system. We're figuring it out. So yes, the correlation tool is a lot better. And I think then you've got to look at too, though the tool can only do so much of that correlation and sometimes you might just need to find out that there's a syntax that you wouldn't think of and they didn't think to put it in their document because it's there, but it's just not. Something that you think about. It's kind of like sometimes people ask me about testing. Certain questions and I don't even remember. But then when they ask it a different way or a different context, you remember. I think it's the same way for the tool vendor is they try to cover everything you need, but they don't. They can't always put it in the document.

[00:25:31] Joe Colantonio Also, I think that's flexibility where you can create your own using code. I know with Load Runner back in the day was C, I assume it's still C. And Web Load is on JavaScript, which I think more developers know and you probably get a lot more help. But so do you see that as an advantage as well? Being using JavaScript over C has been able to make you more flexible or get more help around coding out of the things that are available out of the box?

[00:25:56] Paul Kanaris Probably where I don't pick up on that advantage is I'm also the automation architect, so I write all the code over there. When you ask that question, Joe, I have to go, Do I get help? The answer is no. How much can I pull from we do that all in Java groovy anyways. So JavaScript does give you a lot of the leverages and if you go look at the JavaScript documentation that shows you a lot more information. By going with JavaScript rather than something C-like just remember it wasn't really C so you couldn't really you C either. They didn't go either go C or not. They kind of created their own hodgepodge. This advantage with JavaScript is you can go find the entire JavaScript documentation that'll show you how to use write expressions. Yeah, there's a lot of information out there about how JavaScript works. That's going to give you the leverage that you can do a lot of those correlations without having to ask someone to help you because it's there, it's just you need to have a resource that you can access. I think that that is a big advantage that they are actually implementing a language that is industry rather than proprietary. And that's probably another good thing, Joe, to add to your checklist. What are they offering in the way of the language? Is it something industry standard or is you're going to have to go to their school and learn it? You can always go ahead and do what people did when they wanted to learn Load Runner. They got to come spend a week with me as an instructor and I would show them how to use Load Runner. I have not had to go to Rad view to learn Web Load.

[00:27:41] Joe Colantonio Right. That's true. That's a good point. Another thing you mentioned, I think I glossed over and we shouldn't gloss over. I always think of performance engineering as a team sport where I used to have my database guy on the mainframe person on looking as I was running the load test. But you mentioned something and I'd have to tell him, Hey, I'm seeing something here. Are you seeing it? You mentioned something about Web Load being more collaborative and the fact that you have a browser front end that the whole team can monitor while you're running the test. So they're not all waiting or huddling around your cube. So that sounds like a big benefit as well.

[00:28:14] Paul Kanaris Yes, it being a global company, you really need to build that collaboration a little bit different. One way would be that I had done in the past would be to use something like teams and share my screen. Well, that can cause some limitations, especially when you start getting in the bandwidth. This way you can go ahead and set it up or they can actually go view it. They can see that live run. We can have dashboards that are built. They can go back and forth between dashboards. They don't need to say, hey, Paul, I see what you typed up. The can you show me now the system resources, now that I see what the database is doing. Can you show me the apps or they can immediately go through these dashboards we've built? And look at them real time. They can bounce through them, they can go ahead and start doing some of their own understanding. And if necessary, we can even build specific views for different people. Now, that's going to be something long term we're going to look into. The other part is the way that we've set this up is we're setting it up to where our performance engineering team will be, setting up all the tests. But once they're set up and we have a full regression, we should be able to have those executed by others. You can actually use that browser front end to give them permissions. Now, if you want to start running 5 or 6 performance tests at a time, yes, you do need to buy some extra console license because you'll need them. Does it only lets you run one at a time, but I think that's normally what you would do. Now, a huge mass of companies like an IBM or someone, they would probably need multiples of those. I still think they could save money and also benefit from the tool. As far as its flexibility and the way that it works, like you mentioned with the JavaScript, it can reduce that specialization because I know when I wanted to Load Runner, I had to find people and they weren't necessarily as cost effective as if I needed someone that understood JavaScript.

[00:30:14] Joe Colantonio That's a great point. That's a big point because I used to have to hire people that knew Selenium, that knew how to do Java and how to do automation and Java is really hard to find. It's almost like a unicorn kind of same thing, I guess, for performance engineering, right?

[00:30:27] Paul Kanaris Yes. If you want those things, you got to spend a lot more money. And that was one of the things that made Load Runner a niche. If you knew how to really use it, you could make a lot of money. If you want it, if you didn't get to do that as a profession, but you were trying to do that as part of your job. It was next to impossible. Whereas something like Web Load, I can tell you I've got two contractors that do my scripting. I actually do all the Web Load running and I also manage. I have six managers that report to me. So you can see a lot of things that are going on. If you think about that from a QA perspective, not having to immediately dedicate that resource when you start. Now, what this is proving is it works. We're testing a major database conversion. It is proving out its point. We've already found a number of performance issues when we went from one database to another. And this is talking going from DB2 to Postgres. We found those performance issues. They've been identified. That makes a big savings, but we have not had to dedicate a performance engineer. Now, once we get ready to move to the next phase, we proved out its value. Yes, we need a performance engineering team because now you need people managing a much larger work effort. We're going to be bringing in ten different products that all need performance testing by themselves and together. This gets you in the door to start, but it doesn't go away just because now you've went ahead and proved you need performance testing. You can now take Web Load and keep building it. You can add more generators if you need to. You can say, I need a couple more consoles. All this is extensible as you grow. And I think that's another thing that you want to be able to say. Plus, you can keep all your results as long as you're willing to spend the money. You can store whatever you want. Now, you can also export because we all know with load testing, you run a lot of tests that you throw away. Well, I probably want to keep all my results that matter. I would want to pull those out and I can archive them because I can actually create a zip file of everything from each run that I want to keep for, say, my release report. I can have 6 or 7 runs, put it all together, say put that in the x release folder that becomes part of my artifacts. Or what we're getting ready to do is connected to our test management system that actually is a partner with upload and figure out how to put the results up directly. This is some of the things you want to think about too, is does that tool integrate with other people? Does that may give you another reason why you want this tool over that tool?

[00:33:20] Joe Colantonio Great point. Yeah, because integration capabilities is huge and it sounds like Web Load has it. But I don't want once again gloss over this point. It almost sounds like in your experience, Web Load saved your money because it kept headcount down. I'm not saying replaces performance engineers, but it seems like it helps alleviate the problem having to have niche people that know what just know a tool and that's why you need to spend the money on it, if that makes sense.

[00:33:42] Paul Kanaris Yes. When I first came on board with Load Runner many years ago, we had one person that just did scripting another that knew all about how to create data, another that knew how to do some sort of analysis and understanding of the data. We had to have multiple people. And usually, the other people didn't have any idea how to do the scripting. Now, will we still need someone dedicated to data and how to do data creation? Yes, but I think that they can also have part of their job responsibilities be to assist with that scripting because it is JavaScript. I can now have backups rather than have to have duplicates of, say, the Load Runner scripter. Now I can have my database person who yeah, they can't do the high end, all brand new stuff. But if there's something that goes wrong, they already know how to fix it, they can help work on it, they can be the same ones that can run the test and be able to come up with some preliminary reviews. Now, it may be that they can't do what I do. I can go look at it and I can start just by looking at it go. I think we need to go here and start drilling in. But that's based on 20 years experience. But this gives you more flexibility of how you can use your team. And also another thing where you can locate your team. We're not restricted. Load Runner, you had to have a license for the region that you were deploying in. I'm an international organization. I'm going to be testing products around the world. I also have staff that are around the world. I haven't moved any yet. Right now my contractors are in Bolivia, but I'm looking at long term being able to supplement this with people that are in other countries that have the skills I need because JavaScript is easier to find in some other countries than Load Runner. And I can actually have them working over there. And then they can load it all up on the server, which happens to be in our AWS region and everything runs from here. There's a lot of advantages in, as you mentioned, the staffing. What can I do about it? And a lot of our staff is going to not be because of the total complexity. I told you we're going to be testing 10 to 12 applications. All that's got to be put together. You can imagine just the orchestration of all that. It's going to be huge. You start getting away from the focus of the tool and get to the focus of how am I going to bring all this together to test, number one. And how am I going to get out of it? Meaningful data that we can take action on to either correct performance issues or mitigate the risk of performance issues. There's a lot of things here that we can take and look at with Web Load that are going to give you that advantage. The less time you have to spend either scripting or building things for the tool, the more time you have to focus on the job you're here for. Doing the testing or supporting that testing effort.

[00:36:49] Joe Colantonio Pricing, let's try to tackle this head on. I know Load Runner, very expensive. I'm not telling you to give me a price. Maybe can you give me like a percentage maybe of like how much of a cost to get Load Runner with all your everything you need to compare to Web Loader. It's not even a valid question?

[00:37:06] Paul Kanaris I will put it this way. I called it the Gorilla. And of course it's also a Gorilla. Takes a lot to feed and usually think about how much a guerrilla weighs and figured that out a gold. That is Load Runner. So if you want me to kind of give you an idea, you might be able to go look at a small chimp and put that on the scale for the gold price in comparison.

[00:37:32] Joe Colantonio Paul, you mentioned when you first got there you had a bunch of JMeter scripts. If someone has a bunch of Load Runner scripts and they want to really get rid of the gorilla and go down to the idea chimp sized type, how hard is it to move from, say, other tools into, say, something like Web Load?

[00:37:49] Paul Kanaris They've got to look at one thing. If you've got a huge investment on Load Runner, you've got to make a decision. What's the maintenance cost me in the staffing to continue to grow the gorilla and feed it because I am going to have to take whatever I've done and redo it. It's that simple. You can't take a Load Runner script and magically convert it to Web Load. Sorry that doesn't out there. Anybody who's heard that you can go from VB script to Java, there's a magical tool out there. I will tell them I've got a very big bridge to sell, just like going from DB2 to PostgreSQL. All it is, is take the data out, stick it in. Hate to tell you, it doesn't work that way. So if you've got a big amount of stuff to move, you have to really think about that. Is it worth the cost? Now, from an ease of use, you have to look at you're probably going to reduce your work meantime, from Load Runner to Web Load, probably down to about half the work. And you don't need as high of a level of expertise. Those are things that I would to consider if I had a huge accumulation. But I'd also have to say, am I going to have to rerecord those anyways? Am I making enough changes in my technology? I know simply that I can probably keep my Load Runners. I should say my Web Load scripts around probably for maybe a year, 18 months and I got to redo them anyways because there's been so many changes in the product that they no longer work. They're not like you can do with automation where I can pick things in and out easily. Load testing scripts of any kind are fragile. They go from start to finish and if you start trying to pull them out in the middle, they have a tendency to really do some bad things. That's another way to look at it. If I'm going to change products, is there a way to maybe run them in parallel while I still have a license and then start moving? If I'm going to do that, that would be my way of considering I've got JMeter. It doesn't even do the same thing. You may as well move. Because now you're not getting the same thing that you're looking for. Again, this goes back to what are my requirements of my performance engineering? Do I just want a feel good? The APIs work. Do I want to stay with Load Runner? Because I've invested a lot of ton of money and I think that I can't get out of it. Or do I want to move to a tool that gives me more flexibility and dramatically reduces the footprint cost? I think that's really what you got to think about. That ROI, we all hate those words, but of course, if you can't give them an ROI, companies have a hard time paying for things. If I can cover the loss, I'm gonna take in a reasonable time and give them an ROI then I can do something. But if I can't, I'm going to have to step back and say, like it or not, I'm stuck with Load Runner.

[00:40:47] Joe Colantonio How easy is it to acquire Web Load to get started? Because I know with Mercury they made a pivot from talking to performance engineers like they want to talk to the CTO and so they bypass you completely and sell a huge enterprise. You never got to try it or even engage with it. How was your experience with Rad View and Web Load when it came to that?

[00:41:07] Paul Kanaris Easy go up and download the eval copy. Go have fun. Go play. Probably the advantage I have because I do talk the language and I do know what I'm looking for is I could easily go ahead and answer all the questions that they started asking. Now, also, it depends on your company and also what are you trying to buy? Now, if I were trying to buy 15,000 20,000 virtual users they might wind up have to talk to someone. But I can tell you, they really can't sell into the CTO. They can't sell into the CEO. It's still going to take someone like me that speaks about that. With Web Load, if you're interested in trying to figure out if it's going to work, go try it. Go do a recording. But don't get disheartened If you do a recording, you can't get the correlation to work. Usually what they'll tell you is and this is support ticket, you might not get as quick a response as I do, but they will get back to you because they did with me when we were doing the Eval. They'll get back in the say, hey, try this or they may even do a session with you, especially if you really show that you understand and that you know what you want. And I think that's really where they stand out too, is if you understand what you're looking for and you talk to and communicate, once you do the Eval, I think you'll be able to get the answers that you want. And they also won't try to sell you something you don't need and they'll let you start out where you want. We have a set license right now, but if I need to up it, not a big deal. I make a quick call. I can get that done.

[00:42:44] Joe Colantonio Paul, any other features that have really made Wed Load stand out that we didn't cover?

[00:42:49] Paul Kanaris So I think those are really the big things. You want to look at the ease of use, the protocols that it covers. I really haven't had a chance to look at their AI yet because, like I say, our security people, I talk to them and they're like, no. I had to hold off on that. We're looking at the cloud watch where that just came out, so we're ready to integrate that. And that was part of our AWS move. When I look at it, most of the features we've really covered that I can think of that most people would do because when you start dealing with protocols, when you start dealing with your data, ease of being able to feed data is another big thing. Can you go ahead and use Excel CSV files? Yes, all those are there. How easy is it to add them? Very simple. The data driving is probably another key feature that I would say the simplicity of it. And this gets back to really the focus of why I think Web Load is a good tool for most organizations that are trying to do this is that simplicity and ease of use. You're not have to hire the professional Load Runner Engineer to run the tool. Now, doesn't mean that you shouldn't hire a good Load Runner professional. Should say Load Engineer professional, not Load Runner, but a performance engineer professional. They need to manage the process. You want to give people then that can focus on building out what you're trying to do and how to do it best because they got a tool that they're not spending the time on. I think that's what really makes Web Load stand out, that I can do some of these high-end things that Load Runner did that a lot of the other tools don't, but it's easier. And my budget didn't take the big pinch to where I couldn't afford to do it.

[00:44:37] Joe Colantonio So, Paul, before we go, any parting words of wisdom you want to leave The Guild as it comes or goes to around performance engineering and Web Load?

[00:44:43] Paul Kanaris Well, it's performance engineering as with anything. Make sure that before you start looking at tools, you understand what you're trying to accomplish. Make a list of what technologies you're going to support, what are the processes that you need to work on. And then, like I said, go out there and start looking at. Don't go looking just at JMeter and all the open source. Say I've got an unlimited budget. What is the tool that I would want that will do everything I want? Start there. Once you get what it will do, say, do I need it all or what things don't I need? And then start looking for tools that will do based on that list what you need, but also be ready to look at it and say, Do I really need that? Or was it just I'd like it if something's missing on a feature. Is it could be you don't really use it or you really don't need it. Customers buy things all the time because they think it's neat. Make sure you're getting what you need that will make your testing job more efficient and that you can focus on testing. Don't buy a tool that you have to do a lot of heavy lifting to either understand, make work, use day to day, or you have to spend a lot of time extending. QA, Performance Engineers, all of us need to get over creating all these tools and instead focus on one thing. What am I trying to do to help my customers? And our customers are not the company, they're the people on the other side of the screen of the product that we evaluate. And we want to make sure that from a performance engineering perspective, they're getting what they need because when they get frustrated, they will stop buying your product. They may not do it directly, but if you start costing the company too much time, they're going to be complaining enough to their bosses that you can lose customers. If you're spending a lot of time trying to do that, instead of actually doing the testing you're building, not at all. You're probably not helping that customer.

[00:46:49] And for links of everything of value, we covered in this DevOps toolchain show. Head on over to TestGuild.com/p162 and while you are there, make sure to click on the SmartBear link and learn all about Smartbear's, awesome solutions to give you the visibility you need to do the great software that's SmartBear.com. That's it for this episode of the DevOps Toolchain show, I'm Joe. My mission is to help you succeed in creating end-to-end full-stack DevOps toolchain awesomeness. As always, test everything and keep the good. Cheers

[00:47:23] Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation, testing, and DevOps. If you haven't yet, join our vibrant community at TestGuild.com where you become part of our elite circle driving innovation, software testing, and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges, we're excited to collaborate. Visit TestGuild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve.

[00:48:07] Oh, the Test Guild Automation Testing podcast. With lutes and lyres, the bards began their song. A tune of knowledge, a melody of code. Through the air it spread, like wildfire through the land. Guiding testers, showing them the secrets to behold.