Paul Kanaris
Right? JMeter is very good at doing checks on very small activities. It's when you start getting into the more complex like were a supply chain management system, you're starting to talk about some very big amounts of data that need to be moved around. And again, I go back to what am I trying to do? JMeter was designed literally to say, are my back end calls working? It was not designed to say, how long does that presentation layer take? How much time is that consuming? Where am I really seeing the impact so that I don't get any of those nice phone calls at 2:00 in the morning from my CTO screaming that the system is going so slow that he's got the CEOs all screaming at him. This is what you have to really start looking at is understanding what the tools do, what your goals are versus their limitations. When you go out there, you'll hear, well, you can embed Selenium. Well, that's fine. It's still not check in the return and the responses of thousands of users. Whereas with Web Load, you've got virtual user, these are real virtual users. That's why you can only add about 150 per generator because they start consuming resources. Whereas with JMeter, you can spend up to 2,000 users on one box that doesn't use that much memory. That would be a big savings people say. Well, okay, fine, what am I going to get from it? And that's where I keep stepping back. The other part is how much do I need to do in JMeter to start extending to connect to things like we're going to be keeping cloud watch in from AWS because everything we do is up in the cloud. Well, if I've got to do that in JMeter either need to rely on somebody building that as a plug in or figure out how to do it myself. When I get something like Web Load or other third party tools that are more industry-based, I can get that already and therefore, I save a lot of money and time where I'm not trying to figure it out. Performance engineering shouldn't be focused on building tools, and I call it performance engineering, not testing. People don't understand the difference, but one is running scripts to run scripts. Engineering is developing that test, developing the methodologies and understanding what you're getting out of it so that you can come back and say, Based on what I've executed, here are my potential bottlenecks that I see, and here's where I think we need to start looking. And also adjusting those tests. How do I find out if the background servers are chewing up my system or not? Once I do that, what do all these metrics mean? That's the engineering aspect. If you can explain what you're really coming out with, you're not going to get anywhere. And that can be really hard with something like JMeter, and that's where Load Runner had a lot of advantages and that's where Web Load steps up to the plate, I get a lot of data. I can look at it, I can figure out where I can be, I can build new graphs, and I can do some things like overlaying my transactions against my server response times. I can go ahead and see those spikes and start looking at what's my relationship, and I'm doing this real time. One of the problems with JMeter is, yes, you can bring it together and you can try to pull it all together, but you've got to actually get that data from the server, bring it in and hope the timestamps are right. Whereas, a third party tool, if it's designed correctly, it should bring that data in to the data at the same time. Now when I'm looking at that spike at X milliseconds of the run, I should be able to go check my resources and say, where is my problem? When you've got a large system like the company I have and most companies that will benefit from something like Web Load, I've got about 45 servers. I've got to keep track of, each doing different things. I've got my application layers, I've got my database, I've got Elasticsearch servers. All these things are interacting. Where's your bottleneck? Well, if you can't precisely gather that, and you can imagine bringing in 49 servers worth of metrics. If you had to pull those from each server, bring them in. It would be extremely challenging. Whereas Web Load, it's all pulled down, I can look at what I want, I can remove variables out of there, I can remove measurements, and then I can really look at start focusing what do I think is my problem and present those findings clearly to my development team.