About this Episode:
Do you want to know the value your performance engineering team brings to your company? In this episode Lloyd Watts, Founder and CEO of Neocortix, will share his economics of performance engineer formula to help you quantify your performance testing efforts. Discover what makes up the formula and how to determine the return of your testing, as well as whether you should choose open-source or vendor-based solutions for your team. Listen up!
TestGuild Performance Exclusive Sponsor
SmartBear is dedicated to helping you release great software, faster, so they made two great tools. Automate your UI performance testing with LoadNinja and ensure your API performance with LoadUI Pro. Try them both today.
About Lloyd Watts
Lloyd Watts is the Founder and Chief Executive Officer of Neocortix, Inc., and serves on its Board of Directors.
Prior to founding Neocortix, Dr. Watts was the Founder of Audience, Inc. in the year 2000. From 2000-2005, he served as Chief Executive Officer and raised the first $10M in venture capital financing. From 2005-2011, he was the Chief Technology Officer. From 2011-2013, he was the Chief Scientist. Audience went public in 2012 (NASDAQ: ADNC) and reached a peak market valuation of about $300M, before being acquired by Knowles in 2015.
Earlier in his career, Dr. Watts worked as an engineer at Interval Research Corporation, Arithmos, Synaptics, and Microtel Pacific Research.
Connect with Lloyd Watts
- Company: Neocortix
- LinkedIn: lloyd-watts-5523374
Full transcript Lloyd Watts
Joe [00:02:06] Hey, Lloyd! Welcome to the Guild.
Lloyd [00:02:09] Thank you very much, Joe. Glad to be here.
Joe [00:02:11] Great to have you on. Before we get into it, is there anything I missed in your bio that you want the Guild to know more about?
Lloyd [00:02:15] No, no. You covered it all.
Joe [00:02:16] Very cool. So as I mentioned in the hint show, you've been posting a lot on LinkedIn, a bunch of different topics. One of them that caught my eye was on the economics of performance engineering. So before we dive into that though, what's prompted you to start posting more on LinkedIn on topics of performance engineering?
Lloyd [00:02:35] Well, my company, Neocortix, has a service that we think is going to be beneficial to load test vendors. And so that's my personal interest in the subject, is that it's important to my business. But it was important early on, you know, that I get a good understanding of the dynamics in the industry, who the key players are and what they care about, what's important to them. So I originally started this conversation on LinkedIn, just talking to some of the local, you know, some of the load testing experts, people like Stijn Schepers, Lauri Kasti at supervisor.com, Paul McLean, several others, and was just trying to get an early indication of how they allocate their budgets towards instances that they spin up for load testing compared to license costs of tools that they buy. So my initial question was just about the relative expense budgets for those two components. So I asked this kind of open-ended question, “Hey, do people spend more on instances than they spend on licenses for their software?” And I was surprised and pleased by this sort of flurry of answers that I got back from a wide number of, a large number of people that all came back with: “What about this? What about that?” Like, for example, Scott Moore came back and didn't answer the question at all about the cost of instances and the cost of licenses. And he came back and said, “Well, what about the cost of having your website go down on Black Friday? That's a cost, right?” And so we had this set of contributions to this conversation about what all the different cost components could be. So that was kind of the beginning of broadening my thinking about this. So there's not just the cost that you pay for AWS that you spin up instances on, the cost to your software license providers, the NeoLoad or LoadRunner, or the places where you would spend your money on licenses. There are all these other costs. But some of them, like the ones that Scott Moore was talking about, “Well, they're a cost of not doing performance testing.” That's a different kind of cost. And so that got me thinking about, well, there's these things that are direct costs and then there's other things that are really more benefits, more of a benefit of doing performance engineering, and it becomes a cost if you don't do it. So anything that is a cost, if you don't do performance engineering, I'm calling that a benefit. And then just to kind of close it out, the way that I looked at the end is, okay, you've got a bunch of benefits and you've got a bunch of costs. And generally, people analyze that as a return on investment. If my benefits outweigh my costs, then I've got a positive return on investment. And so now I had the beginning of an economic model of the benefits and costs of performance engineering.
Joe [00:05:26] Very cool. And for the folks that are listening, I'll have an image of this in the show notes. It's a really cool graphic that Lloyd put together that shows this equation that he has set up. So Lloyd I thought we dive into maybe each aspect that make up that equation, see where it takes us. The first section you have, I think is certification. So how does certification play into the economics of performance engineering?
Lloyd [00:05:46] Yes, well, so this is one of the three key benefits of performance engineering: certification, performance, and stability. It’s a concept that originated from discussions I had with companies in the e-commerce space a few years ago, particularly with third-party providers of search engine widgets that recommend products on larger websites. These companies often need to assure their clients—big retail platforms—that they can handle high traffic without failing under load, especially during peak times like Black Friday. Interestingly, in a recent industry workshop, this concept was echoed by a CEO from the UK sports betting sites sector, who emphasized that similar certification requirements are crucial for betting platforms. With high stakes and surges in demand around major sporting events, betting sites need robust load testing to ensure their services remain reliable, attracting clients who demand seamless performance. So, beyond just meeting a technical requirement, certification in performance engineering directly translates to winning business and maintaining trust under heavy load.
Joe [00:07:37] You know, that's probably a benefit, maybe a lot of companies don't think about until they're in a certain situation, I assume. Do you think this is a benefit most companies know about?
Lloyd [00:07:45] Exactly. I mean, until they have a customer, it's not a customer with a gun to your head. It's a customer with a business that you'd like to win. And they're saying, “Well, you want this business, you're going to have to prove that you can handle the load that we expect to deliver to you.” So it's a benefit to the company to do the certification. But the nice thing is that now that's a quantifiable thing. That company could basically say, “Okay, I have this much in revenue that I wouldn't have if I didn't do performance engineering. So we can put a dollar value on that. That's two million dollars a year of business I get from that company that I would lose if I didn't do load testing.”
Joe [00:08:19] Absolutely. So besides the benefit of certification, I think another one was performance. So what do you mean by performance in this particular context?
Lloyd [00:08:27] Yes. So the benefit of performance, that's the second component of the total benefits. So that's a well-known idea that a lot of load testing companies talk about. It's the benefit that you get from having a fast, snappy, responsive website. So the general idea is that for every second of delay that a user has when they come to your website, they're more likely to abandon the page. If you have a really slow, sluggish website, they're going to want to go to another page and buy their shoes over there where there's a snappier, more responsive experience. So there's a translation, there's a scale factor. Basically, every second of delay in response time corresponds to so many millions of dollars worth of lost sales to a large shoe company, for example. So now we have a relationship between delay time in seconds and dollars lost in sales. Again, we can turn that into a benefit of doing performance testing and performance engineering. If you invest in performance testing, if you test your website, you find out how slow it is and then you do the engineering work to improve it. If you improve your response time by two seconds and it's two million dollars per second is your estimated cost of delay, then okay, well, that's four million (two seconds times two million per second) that's a four million dollar benefit you get by improving your response time. So that's a benefit and again, we can put a dollar value on that.
Joe [00:09:51] Very cool. So once you have the dollar value on that as well, I think the next one for benefits is stability.
Lloyd [00:09:57] Right. Exactly. So stability. This is the one that Scott Moore was driving home in the conversation. So his way of saying it was quite sort of sarcastically he was saying, “Oh, you'd rather have a website failure and have to do war room triage after the fact. Oh, those are cheap.” That's what he said. So he's sort of sarcastically making the point that the cost of a website outage on a serious sales day like Black Friday is huge. The huge dollar value in reputation, in lost sales on that day, lost longer-term sales to people who switch over to another supplier, for example. So he's making the point there, again, of a huge cost, but a probabilistic one. You have a cost that you might end up paying if you have the misfortune of having a big failure event that happens under load. So that's a cost that you might pay if you don't do performance engineering. And once again, my philosophy there is if it's a cost that you would pay if you don't do performance engineering, then that's a benefit of doing performance engineering. So I count that again as one of the benefits. That's the benefit of having a stable website because there's a huge cost if you have an unstable website that goes down under load.
Joe [00:11:13] Awesome. So those are the three benefits I think you also have cost then. So I'm curious to know how the cost baked into the benefits. I guess the first one was instances, I believe.
Lloyd [00:11:23] Yes. The cost of instances. Okay, so those are the three components. That was the first one that I was actually directly interested in. That's what made me ask the question in the first place. “How much money are people spending on instances?” Most performance engineers that's spinning up AWS instances or Google cloud platform instances or Azure instances, but spinning up these cloud instances that can act as artificial load generators and then can measure the response times that come back that's the essential operation that's being done in a load test. Well, if you're going to spin up tens of thousands of instances to simulate a million virtual users to simulate a real Black Friday event for a very large company you're going to have to spend some real money on commissioning those instances and keeping them up for an hour or as long as your test is. So my question is, was at the time, “How much money are people spending on commissioning those instances and what's the annual budget for that?” And so I will tell you that no one really gave me a direct answer on that. But what I could infer from the answers that I did get was that depended very strongly on the size of the company. So if you have a small startup that has created a nice website and they have a few hundred customers, then they don't need to do a gigantic load test and they don't need to run it for that long. And the benefit that they get of preserving their good reputation is small because they're not well known in the world. If you compare that at the other end of the scale, so let's imagine you've got a two-billion-dollar company, something huge on the scale of United Airlines or something like that, where the cost of going down is enormous. They might have 50 performance engineers to make sure that their website is fast and responsive. And it's possible they might spend three million dollars a year on spinning up instances and making sure that they're capable of handling enormous load in advance of major travel events, for example. So basically, the amount of money that people spend on instances, loosely speaking, I think is proportional to the amount of revenue that they get and the size of the company. So that's the first component. It's the cost of instances for load tests.
Joe [00:13:31] So cool. All right. So we went over the first cost. It was instances. So how do licenses then play into this equation?
Lloyd [00:13:37] Right. So that's the most straightforward one of all, which is if you are using commercial tools like, for example, NeoLoad or LoadRunner or Gatling Frontline, there's a number of tools like that. But then you're paying some kind of licensing fee to be able to use those tools. And generally, of course, the people that are choosing to use them are doing so because they find that it's well worth it. You pay good money for good tools because they more than pay for themselves. So that's a straightforward one. You have to pay for good software. If you want to use those good tools, you have to pay for the instances that you spin up in order to actually execute the tests.
Joe [00:14:16] Did you have some pushback on this one? I thought I saw a response about licensing that just because something has licenses are commercial, too, doesn't necessarily mean it's worth it.
Lloyd [00:14:24] Well, right. So this is one where there's a number of different opinions. And in some cases, emotions run kind of hot in some occurrences. So basically, there are some people who really love and swear by their favorite commercial tool. And this is really worth it. And I'm really glad that I pay for it. And of course, that's just fantastic. And those tools are very valuable. They did another whole posting on why would you buy these commercial tools? And the answer is, the short answer is because they're totally worth it if you get these benefits of great support, great integration with DevOps, CI/CD pipelines, and so on. There's a number of great reasons, great protocol support, for example, there's a great set of reasons why people would love to use the professional paid tools and be more than willing to pay the license fee. So I have a whole blog post just on that. And that's one of the most popular ones, is singing the praises of the commercial tools. At the same time, when I posted the general comments about comparing the various benefits and the various costs I also got responses from people saying, “Yeah, but wow, some of these free tools like JMeter, there's the free versions of Gatling and k6 and Locust, for example. So there's a bunch of free tools and well, wow, they're surprisingly powerful.” And one person said one of the benefits of the free open-source ones is that you can extend them. So if there's something that you need to do and the big commercial paid tools don't support it, it might be faster for your engineering team to actually modify one of the open-source tools than to submit a feature request to one of the professional vendors and wait for them to do it. So that's one of the tradeoffs that people make. There was another observation that I had, which was that generally I mean, it wasn't universal, but there was a general trend that the people who were favoring the free tools were often coming from smaller companies. Now, that makes sense. If you've got a very limited budget, then the free tools will be accessible to you and you might not be able to afford the really great expensive commercial tools, whereas the larger companies that have huge reputations on the line, they don't want to have their website crash. They have huge revenue. They may have two billion dollars a year in revenue. They can absolutely afford to pay for the tools and get the additional confidence that comes from having these especially well-supported tools. So this question of free tools versus commercial can be controversial and people can take strong opinions about it. I tried to take the position of those arguments to be made in both directions, and I posted in favor of both points of view.
Joe [00:17:14] Absolutely. I guess I was spoiled. I used to work for a large enterprise like you said, and we had all licenses for everything. And I used LoadRunner pretty much for 20 years. And then you go to JMeter and you're like, what the heck is this? So I can see the pros and cons for both for sure. So I guess the next one for cost was I think the last one is engineers. How do engineers then play into this whole equation?
Lloyd [00:17:34] Right. Well, so that's one that I must admit that I wasn't thinking of at the beginning. So as far as I was concerned, the limited view that I had at the beginning of the conversation was the costs that are being spent on a particular load test would be the cost of the instances that you're spinning up and the allocated cost of the license. So if you're doing one big load test a month and you've got an annual license fee, then one-twelfth of your annual license fee would be allocated cost of the software license for that load test. So I was just thinking in terms of money that is spent directly attributable to each load test. But then it opened my eyes when the first comment that came back was from my friend Lauri Kasti, who is the founder of supervisor.com, and he said, “No, actually the cost of the instances is negligible and the only cost that matter is the cost of the licenses” which I had asked about. But then the cost of the engineers, so like the salary of your engineering staff, well, that wasn't even in my mind at the time. I wasn't thinking of that as an expense associated with a particular load test. But from his point of view, that was also one of the costs of doing performance engineering and performance testing is the salaries of the engineers who are on your team who are managing those tests. So, in fact, I owe it to Lauri for extending my view of that. And that's where that third term came from. So then I have the three components of cost, the instances, the licenses, and the cost of the engineering staff. And then we have the three components of benefit, the certifications, the performance, and the benefit of stability.
Joe [00:19:14] So when would someone use this equation and does it help them determine when they should use a commercial tool versus open-source? Is that what this helps them with? Or just in general, why maybe a commercial tool might be more beneficial for them?
Lloyd [00:19:25] Well, yeah. So I think that it does help. However, you could also I think it would be fair to say that most managers that are doing this, making decisions about what tools we should buy, how many…”This year should we hire, more performance engineers or should we pay more for tools or should we do tests more often?” How do they make decisions about what to spend on? As far as I can see, I'm unusual in trying to turn this into a quantifiable thing where I can put equations on it and say, you know, we can calculate what the dollar value of the benefits are, the dollar value of the costs. We can divide benefits by cost and we can get a return. Well, so that's an analytical approach that I take. I think it's fair to say that many project managers wouldn't necessarily do a mathematical calculation like that. They would simply eyeball how much… “Yeah, we need more engineers here because the ones we have can't keep up with the effort that we're asking them to do, or we should spend more on instances because we had a failure last time, last year and we need to make sure we don't have that.” So there are actually two more elements to that first slide about the economics of performance engineering, which were the constraints on it. Essentially, we've talked about the three components of the benefits, but there's another line here that says all of the benefits added together can't possibly be greater than the total revenue that you got.
Joe [00:20:50] Right.
Lloyd [00:20:50] Because essentially those benefits represent revenue that you would lose if you didn't do performance testing. So the benefit can't be greater than the scale that the company is on. A two million dollar a year startup can't claim that they're getting a two billion dollar a year benefit of doing performance testing because they're not at that scale. So the revenue of the company sets the scale for how much benefit could you reasonably claim to be getting by doing performance engineering. So that's sort of sets a limit on how much you can claim you get a benefit for. And then similarly, there's a constraint on the total costs, the total money that you have to spend on instances plus licensing fees, plus your staff of engineers. Well, that's your budget for performance engineering. And now there's a relationship between these two things. You know, no company is going to spend ninety-five percent of its total revenue on performance engineering. That's too much allocation towards that. And it looks to me like perhaps there's a rule of thumb there that people might spend half a percent or one percent of their annual corporate revenue on performance engineering if they are really making a lot. You know it's an online retail business, then they would be willing to spend. If they make two hundred million dollars a year on an online retail business, well, would they be willing to spend two million dollars a year? That's one percent of their revenue on their engineering team and their software licenses and their instances. So there's a scale relationship between the budget that you'd spend relative to the revenue that you have for the scale of the company that you're at.
Joe [00:22:24] And I guess if this all goes back to the first story you told me about how someone how to do certifications. This actually could give them some concrete thing to say, “Hey, you're not doing too much load testing. You're doing just the right amount. Maybe you're right”.
Lloyd [00:22:37] Right, exactly. Yes, that's exactly it. So I think generally a project manager or an executive, the CFO at a company that's at that scale, they're not necessarily calculating a return, as I'm proposing with my Math here but they're doing kind of an eyeball. They're setting a bunch of parameters in their spending relative to what they perceive the risk of not spending it and how much can they afford to spend. And of course, that depends on the current conditions. As you go into a recession, then, well, you might spend less. You might have to lay people off and so on. So those things can vary over time. But in a sense, what I'm trying to do is quantify the eyeballing process that people are loosely doing all the time. You could say the same thing in your own family's personal budget, how much money do we spend on various insurance and food and rent and various other things. And, well, we all make sort of eyeball decisions about that's too much. This is not enough that we're spending on this thing. We need more of the benefit of that thing, for example. So we all make those decisions in an informal way. I'm just trying to put it in an equation so we can look across companies and at different scales and see, is there some sort of rules to it? It's about one percent of the budget. What, one percent of the revenue goes to the performance engineering budget? That's kind of a rule of thumb that seems to come out. In fact, actually, that's that particular thing came from a little case study that I informally did. That was another informal conversation with one of my advisors who was involved in setting up the load testing for one of these e-commerce companies several years ago. And the inside information there was that the company had a thousand employees, two hundred million dollars in revenue, and they were spending eighty thousand dollars a month on AWS instances. So their cost of instances was 80 000 dollars a month, which is 960 000 dollars a year. So approximately one million dollars a year that they were spending on instances, not licenses, not their engineering staff. The cost of instances was about a million dollars a year and their revenue was about two hundred million dollars a year. So that means for them their budget for instances was about half a percent of their total annual revenue. So there's a rule of thumb that came from that particular data point that I had. And it approximately scales if you had a two billion dollar a year company, would they be willing to spend ten million dollars a year on instances not one million? And well maybe they would.
Joe [00:25:07] But yeah, I love this because it gives people in performance engineering or teams to be able to justify maybe why they're doing what they're doing in a way that upper management can understand. If you say, look, we understand this is a cost, but it's actually when you break it down, it's actually a benefit because, in the end, the return is greater than what you're actually investing in this team to help you with performance.
Lloyd [00:25:28] Exactly right in the sense that several of the comments that I got in the LinkedIn conversation were like that. It's like, “Oh, this is formalizing a way for us to justify our budget to our executives.” Well, so most performance engineers are focused on the technology and the software and the techniques and the results they're trying to get. They're focused on the engineering. But there comes a time in the engineering manager's life when he has to go to his boss and talking about the what's the budget for next year and are we going to increase your budget or not? And this is where having some additional tools to be able to say what quantifies the benefit of spending a little more here. And in some cases, look, if we spend it right, we can get benefits that we weren't getting and we might even be able to cut costs at the same time. That's where you get a massive return, an improvement on your return if you can get your benefits to go up and at the same time get your costs to go down. Now, you've got a really winning proposition when you go to the CFO or to the VP of engineering who has to make a budget decision. You've given him a really strong argument to take the engineering actions that you want.
Joe [00:26:41] Absolutely. And I think someone else made a suggestion that people should do proof of concept. So even this helps once you have the equation in place you can also do a proof of concept to make sure that the tool you selected based on that equation is the correct tool. I don't know if they tie in, or is that just an extra type of activity?
Lloyd [00:26:58] Well, so no, I agree with that. So once you have the general equation framework that describes what the benefits are, what the costs are, what's the return, and what are the constraints on revenue and budget then now you can use that framework to do a number of different scenarios. So, for example, the manager of this performance engineering team could say, “Well, look. It looks like we're being given an extra 200 000 dollars so we could spend that on burdened cost on another engineer. And that might give us the additional benefits that we need. Or maybe what we should do is keep the team the same size as it is, but maybe we should use a more high-performing and potentially more expensive software package for upgrade from one tool to another, for example, to get a greater benefit.” So those kinds of tradeoffs or should we do more tests? Should we spin up instances more often, do more instances, do larger tests, and do them more often? So you could do different scenarios, as you just said, and then you could try to quantify for each one of those scenarios how are the benefits affected by that choice. And so that comes now to the couple of the other follow-on posts that I did. I said, well, so let's look at it. Suppose you make a decision to go with a high-performing paid software tool, then how are the other costs and benefits affected? So you make a decision. I'm going to spend more on C_licenses so on that slide the there's a little red arrow. You're spending more on C_instances and that increases your costs. However, if because you've got great software, the number of engineers you need to have on your team goes down. It could be that it's a net cost reduction. Spend more on software, but you don't have to hire quite as many engineers. And so overall, your costs might go down even though the cost of licenses goes up. Another reason this came from an interesting conversation with the folks at OctoPerf. They said, “You know, the performance, the efficiency of the instance utilization is better in the paid tools than the free tools.” They said, “Basically, we've put engineering effort into making our load generator software very, very efficient, more efficient than the free version of JMeter.” Well, that means that okay, if you buy their tool or you pay for a license for their tool, then your cost of instances will go down because you can get the same load with a smaller number of AWS instances. So that's a great example where spending more on software licenses reduces your costs on instances or on engineers. So these are the kinds of tradeoffs where the various cost terms compete against each other or spending more on one reduces the cost of the others. And similarly, also spending more on, for example, the cost of licenses can increase your benefit of stability. So, for example, if you have even better software tools for doing performance testing, you could argue that, and you do the tests well, then our likelihood of having a website crash on a Black Friday event has gone down. So that means your benefit of stability, you will have a more stable website. So paying more on licenses means you're less likely to have to spend a lot of money repairing the damage of a website failure. So, yeah, so I put beside these equations little arrows costs going up are color red. The costs going down are color green. And similarly, benefits going up are green and benefits going down are red. And so you can see, “Oh for this decision I might make these benefits go up and these costs go down or up. What's the net improvement on the return?”
Joe [00:30:40] Nice. Now, Lloyd, you talked a bunch about how you've spoken to all these other companies. Now, you touched on it a little bit, but how does Neocortix actually work with these other companies? Or can you just expand a little bit more on what your company does?
Lloyd [00:30:50] Sure, of course. Yes. So, well, we fit into the ecosystem in roughly the same place as where the other cloud providers fit. So like AWS, Google Cloud Platform, and Microsoft Azure, what they do for the load testing companies is that they provide instances that allow the load testing companies to deliver the load. So they put load generating software onto the instances and create artificial demand for their website and they measure the response times. So the load generator instances are what's being provided by companies like AWS and Google and Microsoft. Neocortix is in the same position in the ecosystem as where they are. We are another way of providing instances, but with some differences. The thing that's common about the big guys like that, the AWS, Google, and Azure, is that their instances are servers that are located inside data centers. And what Neocortix does is we have a worldwide network of mobile phones that are located in people's homes. And we're incidentally, we're paying them for the right to use their processor to do work for our customers. So we have a shared economy business model. We pay them. We're not hijacking their devices. We're doing it with their consent. They rent their processor to us so we can put our instances on mobile devices in people's homes and use them legitimately as load generators for a load test. Well, that's a very different testing scenario than putting all of your instances, all of the virtual users of your load test into servers in a data center. We argue that the Neocortix scenario is much more like a real-world actual Black Friday event. If you think about what happens on Black Friday, it's not just that you get a lot of demand on your webpage. You get demand coming from millions of people, all in their current living environments, it's living rooms in Oklahoma and Boise, Idaho, and New Jersey. People's homes all over the place is where the demand is coming from. It's not all coming from US West Oregon AWS. It's not all coming from one place. It's coming from all over the place. And if you want to deliver the load in a realistic way and especially important, if you want to measure the response times, the most realistic way to do that is to do it using many mobile devices that are distributed all throughout the world. So the Neocortix value proposition is you really should be doing load testing using mobile devices globally distributed and located at the last-mile endpoint so you're measuring the true response time. If you measure the response time to a really high-performance data center, you're going to get an underestimate of how long it takes for the response time to the living room in Oklahoma. I'm using that phrase. I heard that first from Tom Lounibos, who was the CEO of SOASTA, which was another very famous load testing company, performance engineering company. It was acquired by Akamai. When I told Tom what we were doing, he said, “Oh, this is a great idea. I had this idea to make a company to do this in 2011.” That's another story I'll go into in a second. But he said basically that's what you're doing here. The reason this is such a great idea and why he came up with it before I did in 2011, he said, “This is such a great idea because you're measuring the last mile response.” And he said, “The phrase I use for that is you're measuring the response time to a living room in Oklahoma. And that is not a data center in Oregon.” It's a completely different response time. It has the last mile, whether it's on Wi-Fi or not, you have some reduced bandwidth, for example. So if you don't measure it in this realistic way and you use high-performance data centers and high-performance instances for your load generators, you will necessarily get an underestimate of the response time. In the next day, I'm going to be posting another thing that it's going to show that picture very clearly.
Joe [00:34:55] So you were talking about SOASTA and why it was such a great idea. Well, I think it's a great idea as well. It's making sure you're using real mobile devices from different locations to really emulate a proper load, not just from a discrete location all the time.
Lloyd [00:35:08] Yes, exactly right. So it's an important principle and it has implications that are more subtle than might first appear. So basically, just arguing for doing a more careful measurement if you just express it like that, okay well, you know, you could see someone arguing it's not quite as accurate as if you did it right. Doing it with the devices, instances inside the data centers, what's the difference? How big a difference could it be? Well, actually, here's why this is so important. So the importance of having an accurate measurement of a quantity that you are trying to optimize, basically, you've got a very significant situation here where you have a measurement that is inaccurate and in particular is biased in favor of being an underestimate. It's a significant underestimate of the real-world user response times. So if you have a biased, not just a noisy measurement, but you have a biased measurement that is inside a control feedback loop, that's you know all kinds of terrible things happen if you put a faulty sensor with a bias in one direction inside an actionable loop or the actionable loop is the performance engineering cycle. If you're measuring, let's think about that, if you're measuring the response time with your load test and we've got a target. The SLA, the service level agreement target says we want to have a response time of three seconds. And the load test from the data center is telling us that it's 2.9 seconds. Okay, great. We've met our goal and there's no point in doing any further engineering. Okay, well, if that 2.9 seconds that's calculated and measured from the data center load test is a bad underestimate and the real performance is 4.5 seconds mean performance time, then the action of the performance engineer is now the wrong one. Instead of saying, “Great, we've met our target and we can stop doing reductions in response time, we don't have to make our server any faster because it's fast enough. We meet our goal.” Okay, well, you only meet your goal in the presence of this optimistic underestimate of the response time from a faulty test. If you had a true measure of how and what the real user response time is, you'd see, “No, we need to do more performance engineering. We have to remove another second and a half of response time to get from four and a half seconds down to three seconds.” So any time the general principle, any time you have a faulty sensor, especially a biased one inside a control feedback loop, you can get disastrous results. If anyone needs an example of that, the Boeing Max crashes are perfect examples of that. A faulty sensor that said the nose needs to go up when it was already up high enough. If you put that in a control loop controlling what you're doing with the wings, you can crash the plane. So, no, it's a big deal having a faulty sensor inside a control loop.
Joe [00:38:12] Absolutely. It makes sense. Okay, Lloyd before we go, is there one piece of actionable advice you would give to someone to help them with their performance engineering testing efforts? And what's the best way to find and contact you or learn more about your company?
Lloyd [00:38:24] Well, that's a nice invitation. Well, so let me begin with just take a look at these LinkedIn posts about the business model, the economics of performance engineering on my LinkedIn page. So my name is Lloyd Watts. You can search for me and find me on LinkedIn. I think that's it's a worthwhile effort to really understand what are the costs of performance engineering, what are the benefits, and how do they trade against each other. Those relationships are subtle and interesting and worth understanding. I've been surprised at some of the controversy that I've seen about it, and that's why I went to the trouble of making these equations was to make it clear so everybody could see with clear eyes what the tradeoffs are. I think that's helpful. So I'd say, check that out. Also, there's this follow-on set of posts. I've done the first one now on measuring response time, and I'm about to post another one, which is going to make this point about the possible errors from doing load tests in a particular way. So please check that out, too. I think that this is important and it might open people's eyes to something that is common in the industry, data center-based performance tests, which I think could be improved. And we can actually help the field advance by paying attention to the fact that this is not quite optimal yet. And there are things we can do to make that better. And the actionable thing here is for companies to get that right. If they do the performance testing in the way that is most accurate, then they will optimize their performance to the benefit of their customer and they will be in a better competitive position. So if you do it right, your company will make more money. And the flip side of that is if you do it wrong and your competitors do it right, then they will make more money and you will be in a zero-sum game and lose money. So there are millions of dollars on the line here on doing this right, but we have to first understand the principles. What is the inaccuracy in the measurement? And for God's sake, if it's going to be inside a control loop, it better not be biased.
Rate and Review TestGuild Performance Podcast
Thanks again for listening to the show. If it has helped you in any way, shape or form, please share it using the social media buttons you see on the page. Additionally, reviews for the podcast on iTunes are extremely helpful and greatly appreciated! They do matter in the rankings of the show and I read each and every one of them.
