Here are some of the top questions and myth busters regarding test automation using it.
Of course, the very first question asked was around a hot button topic Cypress vs. Selenium.
Cypress.io Vs Selenium
Any time you compare two tools, someone inevitably gets offended.
So, to set the tone, Gleb began by saying that whatever they’ve done with Cypress is because they looked up to Selenium.
Truth be told, we all use Selenium, and we have nothing but respect for Selenium WebDriver and all their other testing tools. With that out of the way, we got into a few pros of using Cypress, and some instances in which using Selenium might be a better option.
Features over Selenium
Here are three features of Cypress that do things differently than Selenium. You can decide which features matter most to your team when choosing the tools that are best for you.
- It runs in the browser, so it allows it to interact with your code, the network, the methods, and even DOM updates in real time. Any tool that's not running in the browser cannot do that as well. So with Cypress, it's an advantage.
- It also has built-in retries. This feature is essential because modern web applications are continually updating themselves, but they're too unpredictable. For example, during test execution, my network might take a long time, so you can’t just expect an element to appear in, let's say, 100 milliseconds. That’s why separate commands always have built-in retry and polling to ensure that even if an element takes a little bit longer, the test will pass, is not flaky and doesn't need hardcoded delays.
- Cypress also concentrates on easy installation. With Cypress NPM, you can simply install a single command and get all the screenshots and video by default on every CI, on every platform. Ease of installation is definitely there.
On the other hand, Selenium has a couple of advantages as well:
- Selenium supports cross-browser testing. With Cypress, it's just on the road map, but Selenium already has it. So both have advantages and disadvantages, and you should evaluate both systems and decide what works for you.
Currently, Cypress.io Browser Support is for Chrome browsers only.
Another topic some folks had questions about was the difference between how Selenium and Cypress interact with the browser under test.
One concern that came up was how Cypress interacts with a browser, and whether it’s really driving the browser as a real user would. That’s something Selenium would find that Cypress wouldn’t.
For example, is there a difference between telling the browser I clicked on something versus emulating a real user clocking through a browser?
But Cypress does have native events on its road map and will have support for them soon. That will enable you to do things like file upload, download or copy/paste onto your clipboard. You’ll still have the synthetic events, but you will also have native events for all interactions.
That update, when it happens, should remove any concerns you have about the browser interactions not behaving the same as Selenium.
When creating automated tests, you sometimes need a way to mimic the behavior of a real service. This can be achieved using a mock object.
Can You Use BDD with Cypress?
Another approach some teams have adopted is behavior- driven development. Can you achieve a BDD approach using Cypress?
While Cypress doesn’t have an official BDD built into its out-of-the-box option, there is a community-created BDD Cucumber processer that can help.
The NPM Cypress-Cucumber-Preprocessor plugin allows you to write your tests in BDD Cucumber Syntax, and it will then automatically translate them into Cypress.
There’s also a pretty good working sample you can check out on Github.
Cypress for Visual Validation?
You may also be wondering whether Cypress can make full-screen image comparisons against baselines in a way similar to what the Galen framework does with strict layout design requirements.
Cypress does have a built-in screenshot feature, so you can take a screenshot of a full page, a particular element or the whole thing, including the TestRunner UI. They recommend you take screenshots of a specific element so that if it acted differently, you would know which part of the page was different and wouldn’t compare, for example, a dynamic part all the time.
Worthy of note is that there is no built-in image diff in Cypress. They give you a screenshot. But it's up to you to compare.
Luckily, some folks have written plugins that actually do that.
For example, companies like Percy.io and Applitools both have plugins for Cypress that do excellent image diffing.
Cypress.io API Testing
Another common requirement for end-to-end testing is the ability to interact with an API. Fortunately, Cypress supports this as well.
When asked if REST testing was supported, Gleb recommended that folks check out cypress-example-API-testing on GitHub, which contains an example of using the E2E runner to also test Rest and other APIs.
Those are the top questions the Guild had for Gleb. What are yours? Let me know.
A few folks mentioned that they had a hard time understanding the audio in my interview with Gleb. So here is the full transcript so that you can read while you listen.
Joe [00:00:01] Hi Gleb. Welcome to the Guild.
Gleb [00:00:02] Hi. How are you?
Joe [00:00:04] Great. It's awesome to have you. This seems to be a lot of excitement behind Cypress and also a lot of misunderstandings of what exactly it is and how it compares to other tools, so why don't we jump right into it.
Joe [00:00:19] First question “According to you, what are the three features of Cypress over Selenium?”
Gleb [00:00:29] I just want to start with saying that whatever we've done with Cyprus is because we looked at Selenium and we all use Selenium, and we all have nothing but respect for Selenium WebDriver and all other testing tools. So I will say that the three feature of Cypress that does things differently is that 1) It runs in the browser. So it allows it to interact with your code, the network, the methods even DOM updates in real time. Any tool that's not running in the browser cannot do that as well. So with Cypress, it's an advantage. 2) We also have built-in retries we thought that for modern web modern applications are continually updating themselves but they're too unpredictable. My network might take a long time so you can not just expect an element to appear let's say in 100 milliseconds so separate commands always have built and retry and polling to make sure that even if element take a little bit longer, the test will pass, is not flaky and doesn't need hardcoded delays. 3) And also we do concentrate on easy installation with Cypress npm install a single command nothing else to install, and you get all the screenshots, all the video by default on every CI on every platform. So ease of installation is there.
Joe [00:02:42] It's a great answer. A lot of times whenever you talk about a tool other than Selenium so many people think you're talking bad about Selenium. We are not talking bad about Selenium; this is just an option. More options, the better every team is different, and one tool may be better for your organization than another. Not, because the tool is superior, but that's just how you to your team work better with a particularly tool. OK, we got that out of the way :).
Joe [00:03:04] Next question is, let's go to it. It's from Harry. It's been a big topic in this conference so far AI solutions — so can Cyprus currently or is it planning to utilize any AI machine learning to help prevent flakey tests? With the number of AI tools now in the market. It's a must for us.
Gleb [00:03:20] I think you can kind of understand the question in many possible ways.
Gleb [00:03:26] I kind of mentioned the built-in retries that prevent flake because of the unpredictable nature of network and storage. So there is that, and it doesn't require any ML/artificial intelligence. If you're talking about updating selectors and which selectors should be used when you're writing end-to-end tests we have the best practices guide you can use specific data attribute, so you don't depend on the Dom structure or the classes and at the same time if you want to go full way to ML helping you maintain your tests and update your selectors we are talking to companies like Mabl. Maybe something will come out of it. But we're also evaluating statistical methods that take the history of your tests and retry a failing test automatically, so a single failing test out of a thousand doesn't stop the build, but still doesn't get lost when its retried and you can go back and see which tests sometimes fail and have to be retried. So we'll evaluate every possible tool to make your test less flaky.
Joe [00:04:38] Next questions from Nathaniel. He wants to know Because Cyprus is interacting with the web app directly. Is there a layer of code, example risk above that, that Selenium would test, but Cypress is skipping? For example, would there be a difference between telling the app that I clicked versus emulating a real user clicking through the browser?
Joe [00:06:26] My next question is, can we use Cypress.io with a BDD tool like Cucumber?
Gleb [00:06:34] So the syntax of Cypress is very different. It's declarative. It takes getting used to. But people wrote BDD cucumber to cypress tree processor which you use as a plug in to Cypress test and it that case you can write your test in BDD Cucumber Syntax, but they'll be automatically translated into Cyrpess. It's not our official plugin its something that came from the community.
Joe [00:07:04] Can Cypress do full-screen image comparisons against baselines in a similar way that the Galin framework does with strict layout design requirements. It would be a massive help if these tests would run directly from dev code, not a separate test.
Gleb [00:07:20] It's absolutely a wonderful question, and visual diffing is a great technique controller of the page looks the same. OK, so Cyprus takes screenshots it's a built-in feature. You can take a screenshot of a full page or a particular element or the whole thing including the test runner UI we actually recommend you take screenshots of a specific element so that if it acted differently you know which part of a page is different and don't compare for example dynamic part all the time. But there is no built-in image diff in Cypress. We give you a screenshot. But it's up to you to compare. Luckily people have written plugin that actually do that. Two companies that do that Percy.io and Applitools have their own Plugin for Cyprus that do great image diffing. We don't have to do anything we provide a platform.
Joe [00:08:34] What is the suggested migration strategies for page objects to app action migration.
Gleb [00:08:40] I would pick the test that takes the longest because of the painful test. It's going for a lot of action, and they all take a long time it might make sense to be mored into app action. For example, if you are logging in thru the UI and that takes a long time. Maybe you can log in by calling the app directly with username and password, and it may make an API call to shorten that wait. So my migration strategy would be to find a pain point usually an extended running test and see if some parts of it are better to be a single path app action.
Joe [00:09:23] Next question is from Christina. Christina wants to know, “By using app actions, you are not touching the UI. How are we sure that the functional elements work for example, they can actually click on an element or type in a field. Isn't this proven to error?” I think you touched on this a little bit already.
Gleb [00:09:41] You still touch the UI, and you go through a page like a user if you were testing that feature. But when I was doing a testing routing, and I needed a couple of to do it I didn't add to do it again for a bit. Instead, I default app action, and I immediately went through the depth, and without it, you still cover everything you don't repeat yourself. Instead, you use app actions to avoid going through a page again and again, so you feel will cover everything.
Joe [00:10:13] Next question is from Harrison Lee. Harrison wants to know “How do I use Cypress to test some web app in different test environments like dev/QA/production with different base URLs.”
Gleb [00:10:30] Excellent question. So in Cyprus where a lot of config value like base URL and you can set that config in Cypress JSON configuration file or provide it as a command line argument or an environment variable or even programmatically you can change it so you can point at a different base URL by just specifying on command line or setting environment variable and that way you can run the same tests against Localhost or against staging or even against production. We have a lot of examples of that in our documentation. If you search for that, you'll find it.
Joe [00:12:33] Can Cypress.io be configured to run with different browsers to run a test in parallel?
[00:12:38] We have a built-in , but it's not paralyzed on your machine. It's not like a single Cypress instance when you're running locally on multiple browsers it's not going to work. First of all your computer will spin out of control because Cypress is driving your app really really quickly for the test to finish so rapidly your MacBook Pro or whatever you're using will actually just become very hot from CPU usage. Instead, we allow you to parallelize your test and run multiple tests on multiple machines. So if you have a CI run, we have a service that will load balance all the tests and all the tests will be executed on multiple machines in parallel. But it's a different tool from running various browsers locally so you can follow our documentation and if the Paid, unfortunately.
Joe [00:13:35] It's great that this allows developers adding test to their code. But as a QA automation engineer, the only access we have is to the pages and not the application source code. Do you agree that end-to-end tests with pages and Page objects still has a place even with this tp technique?
Joe [00:15:09] Next one is from Staz he wants to do. Do we have to pay for the Cypress dashboard service manage 500 test recordings to have test paralyzed and CI integration in Cyprus or can I get away with the free Cyprus testing or in this case.
Gleb [00:15:30] Test parallelization is a paid feature. That's how Cyprus actually makes money. The test runner is opensource, and we give it away for free. It's not future limited. Half of our small in person company is just dedicated to working on the free test runner. For us to actually give that away for free and keep development going we have to make money, and we make it on private projects, and parallelization is one of the features we can charge for. But soon if you have a genuinely opensource project that means non-profit open for library or web application or framework. We are going to release an opensource license where you can get all the feature for Cypress dashboard for free.
Joe [00:16:20] My next question is do you have a Java binding for Cypress.
Joe [00:17:18] Can Cypress do remote execution like selenium grid.
Gleb [00:17:22] Not yet. So when you run in CI, you can paralyze and load balance multiple machines but each one will still be in its own browser right there in the container and will control it. You cannot control remote browser yet. We are in talks with all the companies that provide like device farm or browser. So stay tuned is just our architecture where we have to close the control and launch the browser is not. Conducive to doing that just yet out of the box and that will require technical work.
Joe [00:19:06] Any multiple browser support?
Gleb [00:19:10] We're doing native events which are a prerequisite for multiple browsers. So in the next two quarters, I would say.
Joe [00:19:19] What browsers does Cypress support as of now? Is it just Chrome.
Gleb [00:19:23] Currently its just Chrome
Joe [00:19:45] Well Cypress work with older ASP dot net MVC applications?
Gleb [00:19:51] anything that actually loads in a browser you can test it. As long as it produces something your chrome browser can load. It's going to work.
Joe [00:19:58] Is it possible to test windows native desktop apps with Cyprus by using a combination with win app driver.
Gleb [00:20:04] Now it's very different architecture. We focus only on testing web apps.
Joe [00:20:09] What kind of reporting tools does Cypress separate support.
Gleb [00:20:12] So you can generate xunit, junit. Mock awesome. JSON report anything that Moca can create– we can create.