When you think of Netflix the last thing you probably think about is automation testing.
But that is about to change.
Why?
Netflix has just open sourced an exciting new tool for testing applications called SafeTest. Created by Netflix Senior Software Engineer Moshe Kolodny, SafeTest aims to provide a pragmatic, flexible testing solution that combines the best features of end-to-end testing tools like Cypress and Playwright with component-level unit testing.
INDEX
Following is some of what I learned from Moshe after interviewing him on my Test Guild Automation Podcast. So this information is sourced directly from the expert.
What is SafeTest?
At its core, SafeTest acts as “glue” or “wrapper” to coordinate between a test runner like Jest or Vitest, a browser automation tool like Playwright, and a front-end framework like React. It allows you to write tests that run both in Node.js and in a real browser, enabling realistic and comprehensive test coverage.
SafeTest works by building the same test tree in both Node.js and the browser.
When a test is executed, SafeTest opens a browser and mounts the component being tested. Developers can then interact with the running application just like an end user would.
Benefits of Using SafeTest
Key benefits of SafeTest include:
- Running tests in a real browser for maximum fidelity, but maintain the ability to mock services and perform component-level testing
- Easily testing authenticated workflows and applications behind corporate proxies
- Automatically getting videos, screenshots, and tracing on test failures to aid debugging
The flexibility to test however you want, without opinion - Allows the testing of traditionally hard-to-test scenarios, like components deep in the UI tree. or hard-to-trigger error states
- Framework agnosticism—works with React, Vue, Svelte, Angular and more
- Easy setup and authentication hooks
Debugging with SafeTest
SafeTest provides a rich, out-of-the-box debugging experience. It includes features like video recordings, trace viewers, and a component explorer that lets you interact with individual components in a failed test. This can greatly aid in identifying and fixing issues.
With SafeTest's detailed failure artifacts, you can spend less time reproducing bugs and more time fixing them.
The Power of Overrides
One of SafeTest's unique features is its “override” capability. Overrides allow testers to pragmatically mock out parts of the application, from individual props and service responses all the way up to the network layer. While not meant to be the first tool to reach for, overrides provide a powerful escape hatch for testing tricky scenarios that might be difficult to set up with traditional mocking approaches.
Consistent Cross-Environment Testing
SafeTest can run its browser in a Docker container, ensuring consistent results between local runs and CI environments.
This is especially useful for visual regression tests that can be thrown off by subtle rendering differences between operating systems. With SafeTest, you can have confidence that your tests will behave the same no matter where they run.
Testing Hard-to-Reach States
By being able to mount individual components in isolation, SafeTest makes it easy to test states that may be difficult to trigger in a full end-to-end flow, such as error boundaries or edge case API responses.
This ability to surgically test any part of your application leads to more comprehensive test coverage and fewer bugs slipping into production.
Advanced Scenarios with the “Bridge”
For advanced use cases, SafeTest establishes a two-way communication channel between the Node.js test runner and the browser. This “bridge” can enable some powerful scenarios, akin to an Electron app. While likely not needed for most tests, the bridge is there when you need to push the boundaries of what's possible with browser testing.
HTML Reporting
SafeTest publishes an HTML Test Reporter that can be used to view the results of your tests. The report includes:
- Trace viewer of each test
- Video of test execution
- Ability to open tested component in the deployed environment
A Pragmatic Automation Testing Philosophy
Moshe emphasized the importance of being pragmatic and iterating quickly when starting with testing.
SafeTest aims to support this by providing a versatile tool that can adapt to a variety of testing strategies without being overly opinionated.
Whether you're a developer looking to test drive your latest feature, or a QA engineer building a comprehensive regression suite, SafeTest has you covered.
Pros of SafeTest vs Cypress
While Cypress is a popular and powerful testing tool, it has some key limitations that SafeTest aims to address. Cypress tests are written entirely in the browser, making it difficult to test front-end code in isolation from backend services.
SafeTest, on the other hand, runs your test code in both Node.js and the browser, allowing you to decide what level of mocking and isolation you need. It provides flexibility to test at the network layer, at the API service layer, or on an individual component, all while running the app in a real browser.
But one of the cons of Safetest vs Cypress is that if your looking for a pain free all in one install like Cypress — SafeTest can't compare.
Cons of SafeTest vs Cypress
Based on my interview with Moshe he freely admitted that the setup process for SafeTest is more complex compared to Cypress.
Moshe mentioned that Cypress has a great developer experience and is easy to get up and running, especially for developers who are new to testing.
In contrast, setting up SafeTest requires more configuration and integration with your application's build process.
This is because SafeTest doesn't compile your code like Cypress does.
Instead, it assumes you have a bundler like Create React App and leverages that to work with your application.
How to Install SafeTest
To set up SafeTest, you need to:
1. Modify your application's entry point to bootstrap SafeTest
2. Use lazy loading and glob imports to ensure SafeTest doesn't increase your production bundle size
3. Configure your bundler (e.g., Webpack or Vite) to handle the glob imports correctly
Moshe acknowledged that this setup process can be tricky, especially when it comes to configuring your bundler. He mentioned that you might need to “dip your toes into Webpack config territory, which is never a fun place to be.”
So while Cypress provides a streamlined, batteries-included experience for developers to start writing tests quickly, SafeTest requires more upfront configuration and integration with your application's build process.
This additional complexity is a tradeoff for the increased flexibility and power that SafeTest provides, but it does make the initial setup more challenging compared to Cypress, particularly for developers who are less familiar with their application's build tooling.
When to use SafeTest
According to Moshe, SafeTest is suitable for 90% of testing scenarios, especially for developers looking to gain confidence in their front-end code. While Cypress's fluent API may be preferable for some, and pure unit tests may run faster for certain edge cases, SafeTest aims to be a pragmatic, go-to solution for application testing.
SafeTest Netflix and Chill – Check it Out for Yourself!
The fact that SafeTest gained over 1,200 stars on GitHub within a month of its release shows the excitement and interest around this new tool. With an active community of contributors forming and an open-source codebase, SafeTest is poised to become a major player in the JavaScript testing ecosystem.
For developers and testers looking for a powerful, flexible testing solution, SafeTest is worth checking out.
I believe it brings a whole new meaning to Netflix and Chill – so give it a try and tell me what you think in the comments. below.
