During the past month I've interviewed both Marcus Merrell and Christian Bromann about a cool new Sauce Labs feature that helps with front-end performance applications.
Here’s what I found.
When talking about performance testing, most folk think of stress and load testing, which involves measuring the response of your back end systems.
There are already a lot of great tools you can use to load test your applications, but Sauce's solution was designed specifically for front end performance testing.
Why is this needed?
A host of modern applications now are using JavaScript frameworks like React and AngularJS front ends. These feature-rich front ends add a new layer that needs to be measured.
So the measurement of the rendering time of an application that is under test on a client's local machine becomes critical.
What Kind of Performance Data can You Get?
Sauce is able to access additional parallel connection to the Chrome browser. They do this by being able to control the communication between the Chrome Driver and Chrome itself.
This allows them to start tracing the browser and get all the information about whatever is happening in the browser engine from Chrome.
They then use the tools that are provided by Google like Lighthouse to make sense of all this data.
Because the tracing log data can have massive payloads (sometime as large as ten megabytes) without using something like Sauce you are probably not able to use them.
Sauce scans the logs for you to easily read the performance metrics.
What Can you Measure with Sauce Performance?
The new Sauce performance feature captures virtually all of JavaScripts rendering metrics, including:
- Performance Score – Provides a weighted average of performance metrics, which is useful in tracking team progress.
- Page Load Time – Indicates when the page was completely loaded (or when the browser officially fired the onload event).
- Speed Index – The visual completeness of the page as defined by Google.
- Estimated Input Latency – Input responsiveness is a key factor in how users perceive the performance of your application.
- Time to First Byte – The amount of time it takes before the first byte of the response is received from the server, relevant application caches, or a local resource.
- Time to First Paint – This is the point at which pixels are first drawn on the scene. This metric excludes the default background paint but does include any user-defined background paint.
- Time to First Contentful Paint – The first contentful paint is triggered when any content (an object defined in the Document Object Model) is painted. This could be text, an image or canvas render.
- Time to First Meaningful Paint – The first meaningful paint is a browser-supplied metric that measures how long it takes for the most meaningful content to be fully rendered on the site. It is equivalent to the first contentful paint, and describes the duration until the browser first rendered any text, image (including background images), non-white canvas or SVG. This includes text with pending web fonts. This is the first moment when users could start consuming page content.
- Time to Interactive – The number of seconds from the time the navigation began until the layout is stabilized, web fonts are visible, and the page is responsive to user input. The page is stabilized if there was no task blocking for at least 50 ms.
- Time to First CPU Idle – How long it takes for the CPU to process all of the page information.
- DOM Content Loaded – The DOM content loaded event is fired when the initial HTML document has been completely loaded and parsed, without waiting for stylesheets, images, and subframes to finish loading. A very different event load should be used only to detect a fully-loaded page. It is a popular mistake to use load time as a performance metric, when DOM Content Loaded would be much more appropriate.
(* The reference for these definitions were taken from Google Lighthouse Docs and FireFox Developer docs.)
Although server-side factors influence these metrics, if you can isolate them and measure them over time, you can capture these deltas.
This should give you an idea of how quickly your application is loading and rendering.
You can also write code that asserts you don't exceed your benchmarks from some previous time.
It also gives you the performance score based on the lighthouse calculations, which is data that you would usually be unable to get with basic Selenium WebDriver.
How to Add Sauce Performance to Selenum Tests
To begin capturing performance values in Sauce Labs, all you need to do is add two new capabilities to your existing WebDriver tests.
- extendedDebugging: true,
- capturePerformance: true,
We Didn’t Have Fancy Sauce Labs Back in My Day
To do this back in the day, I would have to create a WinRunner script. I then had to add a loop timer function that waited for an element to appear in the application.
I would then add it to a LoadRunner controller and place a load against the system using the HTTP Vuser protocol. The WinRunner script would then run on a client machine as a GUI Vuser and keep the timing of the local rendering time.
This, of course, was not accurate, but at the time it was the only way I could achieve this functionality. If I’d had Sauce Labs back then I could have saved myself a bunch of time.
Why this is Cool
Marcus says the thing he's most excited about is the fact that it's not specifically related to load and stress testing; in fact, it is deliberately irrespective of them.
When encountering cases where there is performance degradation, we inevitably start to feel it in our users’ behaviors.
At that point, the first thing most testers do is run a load and stress test thinking it’s a performance problem. But the behavior occurs regardless of whether you're just warming up with a few nodes or running 500 of them.
It's not a linear growth; it remains static.
This may cause you to say “Oh my. That means there's something in the UI that has changed.” For instance, someone has introduced some DOM element that takes forever to run, or someone has increased the size of the images, etc.
We tend to run a load and stress test when we see a performance issue, not understanding that it's actually a UI problem.
With this tool, you can avoid the overhead and use it to help not only diagnose UI problems, but dial in performance on a page that you particularly care about and then notice over time whether the performance on that page changes.
Why Should You Care About Front-End Performance?
The example that Marcus usually brings up is based on his past experience working at RetailMeNot. It's a very Google Search Engine Optimization (SEO) intense B2C Web site that has millions and millions of hits per day.
They had something like ten thousand keywords where they were ranked number one.
Then one day, Google introduced an algorithm called Panda that changed how Google indexed the way their search bots looked at web content.
Before the change, they looked at a combination of things like bounce rate and relevance. With Panda, they started weighing page performance way higher than they did with the previous algorithm.
Suddenly RetailMeNot started dropping in all of their rankings.
It took months of working thru lighthouse to check to see how their page performance was.
Marcus believes that if they had had this feature in place at the time, running alongside their Selenium scripts, they would have known instantly when their page loads started slowing down, and they would not have been hit with this massive SEO penalty.
So if you're in retail or banking or some BTC where SEO is vital, having this running alongside your tests is invaluable.
Speedo
Chris told me about one other feature based on performance called Speedo.
Speedo is a NodeJS package that gives you the ability to run performance tests on Sauce directly from the command line.
Speedo was created to help teams to get started with performance testing, so it was designed for ease of use and takes care of the complexities that are associated with running front-end performance tests and validating the performance regressions.
Speedo easily integrates with any CI/CD system, which is a great way to plug Sauce Performance into your CI/CD pipeline and start capturing performance.
If you have an enterprise plan with Sauce Labs and are using Chrome (as of now, performance metrics can only be captured on Google Chrome), you need to check out this new feature.
