One of the most misunderstood parts of performance testing is load testing. Most folks think all performance testing is load testing, but that’s incorrect. There are are many types of tests that make up “performance testing”.
Before we get into things to consider before performing a load test, let’s go over some basic info on what a load test actually is.
The definition from an old Mercury Interactive LoadRunner workbook I have reads:
“A load test is many concurrent users running the same program to see whether a system infrastructure handles the load without compromising functionality or performance.”
Leandro Melendez (AKA Senor Performo) likes to explain load testing as:
“A load test is a specific sub-family of a performance test. Sometimes I say that performance testing would be an equivalent of Microsoft Office and a load test is Word. Word just a component of Office. Word is heavily used, together with Excel. They are probably the two most heavily used components of Office. But it's not Office.”
Now that we are clearer on what load testing is, here are nine strategies to think about when preparing to do a load test. (Most of this info was gleaned from a 2017 PerfGuild session by Shlomi Nissim, who was an R&D Manager for HP's performance testing tools at the time.)
1. Targeting the Right Workload
First, don't test your application on a large scale without actually needing to do so. There’s no need to stress your platform beyond what is actually expected.
Sure, it's always good to produce higher traffic loads than expected to stay on the safe side, but to be realistic and productive, you should focus on evaluating the right workload that your app will experience in production.
Of course, your website or app may experience demand during more than one peak or spike load time depending on seasonal events unique to your industry. But it is recommended that you begin load testing by emulating a normal day’s throughput first before you focus on testing your peak times.
The keyword here is throughput, which is another commonly misunderstood performance testing topic. Be sure to check out my post, Performance Testing Basics What is Throughput to learn more.
When coming up with your workload transaction testing profile, think about:
- On average, how many operations occur in a typical hour. How about at peak?
- What is the goal of this test?
- What does your server or database activity typically look like?
- When choosing tasks to emulate, are you focusing on the ones that are the biggest risk to your business?
- How heavy if a load do you really need?
- What metrics will you track or are required
Spend some time getting this step right since it is the fundamental cornerstone to creating proper load tests.
2. Load Generator Readiness
Make sure your load generators are ready to sustain a workload. If you’re not already aware, “load generators” are the computers that run your virtual user tests. A virtual user is a script that acts like a real user would when making simultaneous requests to your application & systems under test. This is usually done by emulating browsers communicating with the backend of your system.
Here are some things to consider:
- Remove any software that is not needed from your machines.
- Verify the generators are connected to the network and have enough network bandwidth, etc.
- If you're going to run many virtual users, you’ll need many load generators. Reliable connectivity between the different components of the testing tool that you are using is mandatory for the success of the test.
- When it comes to large-scale tests, it is recommended to allow the load generator to have some sleep time. Shlomi recommends allowing as much as possible because if you have thousands of virtual users running on the same load generator it’s a pretty intensive operation.
All this is important to understand because there’s always a risk that if you over-utilize the load generator itself, you can damage the reliability of your test results.
3. Scripting
It's not enough just to create a test that will mimic real-world scenarios. You also need to ensure that your scripts are not overloading the test tool itself.
Make sure you have optimized the settings for your test scenario. Time, duration of run, monitors selected and amount of info logged—any of these can put stress you’re your tool.
Take any hardcoded or dynamic data that needs to be parameterized into consideration. You don't want to code a test that generate unrealistic failures because of a poorly written script or improper correlation.
Be sure to generate the right amount of data, which will enable successful tests at the end of the day.
4. Think Time
Think time is an essential part of the script logic.
All tools should have some logic of how to add think time by specifying how many seconds you want the virtual user to wait.
Think time is useful to mimic the right workload according to the real behavior of the virtual user. Just like a functional Selenium test, it's also advised to use think time rather than a sleep function.
Not utilizing think time correctly is another common performance testing mistake.
People either forget to add them or else they use two milliseconds. You really need to create a realistic performance test scenario that emulates how a real user is going to interact with your product.
You should look at the tool and ask yourself, “Okay — is it going to take five seconds to make a decision on this page? Is it going to take a minute to perform this task?”
This can also cause people to create unrealistic results that aren't valid, which will, in turn, cause the team to panic for no reason, so avoid this mistake at all costs.
5. Ramp Up and Ramp Down Time
It's probably unnatural for an application to get so many virtual users logging in at the same time and out at the same time.
If you can afford to make the ramp-up and ramp-down time longer, then do so. It’s a good practice to have a ramp-up time when running a load test, as it will take some time before your test reaches a steady state.
Response time and other measurements should only be taken between the ramp-up and ramp-down time frame.
6. Monitoring and Diagnostics
Monitoring and diagnostics also need to be taken into consideration when it comes to large-scale load tests.
Capturing deep diagnostics info during a normal test means that a lot of data will be captured. But for an extended test with many virtual users, the data captured can be staggering. The side effect of this can be that your test tooling might have a challenge trying to cope with all this information.
The same goes for performance monitoring.
Shlomi mentioned that he's seen many customers and users selecting seemingly random measurements without knowing what each counter does, or if they even need to monitor it.
Generally speaking, it's not a bad idea to capture as much monitor data as possible, but to minimize the amount of test data being collected it’s crucial to choose only the essential performance counters.
When performing a load test, save yourself some headaches and stick to using the counters you know how to interpret and use later on to troubleshoot and isolate your performance issues.
7. Workload Scheduling
Workload scheduling refers to how fast you plan to ramp up and ramp down your virtual users.
What will the impact on the load generators be? What is the effect that it's going to have on the application or applications itself?
These are things you need to be thinking about as well when determining a workload event. You don't want to create any artificial bottlenecks.
8. GUI Virtual Users
What happens when you're going to run GUI virtual users?
Typically, GUI virtual users, as opposed to HTTP-level virtual users, consume more resources like memory and GDI resources.
GUI users are heavier virtual users, and their footprint is more substantial. It's critical to know how many GUI users you can run on your load generator.
You should be ready in advance to avoid redundant failures and to prepare the right amount of load generators, balancing the workload across them.
9. Analyze Test data
Be prepared to analyze your load test data.
Remember, the longer the test, the more considerable the number of captured events during the tests, and the more challenging it's going to be to analyze it— regardless of the tool you’re using.
Load testing generates a lot of data.
It's not easy to dig into the test results and find everything you need. Even if you have an easy way to analyze the data (or it's automated), it’s still a pretty challenging process.
You need a plan on how to handle this.
Keep in mind that faulty analysis of your software data will produce faulty results. Don’t jump to conclusions without properly analyzing the data. It takes experience and skill to extract relevant conclusions from the data produced by a load test, so don’t take this step lightly.
Load Testing—Key Takeaways
Before starting a load test, you need to target the right workload traffic.
Identify the virtual user's footprint in order to prepare the right amount of load generators and the distribution of the virtual users among them.
You need to create the data that the virtual users are going to use, and you need to prepare the environment, hardware, servers, whatever it is, the application under test, whether it is the testing tool itself. This holds true for any performance test performed using tools like JMeter, K6, loadNinja etc..
Remember: since large-scale load testing is not a trivial exercise, you should have all the right stakeholders available to help support and analyze things as the test is running and to determine optimization actions.
Resources
* Load Testing for eConfidence (Seque)
* LoadRunner for Web 7.0 Student Workbook
Performance for Everyone
Lastly, I believe performance testing is not something that should only be accessible to a few but instead should be accessible to everyone.
That's why I create the online conference PerfGuild over 4 years ago.
So do you want to really take your performance testing to the next level?
Make sure to check out and register for our next PerfGuild an online conference dedicated 100% to helping you succeed with all your performance testing efforts.
