Cypress.io Top Automation Questions (Hot Seat)

Gleb Bahmutov, VP of Engineering at Cypress.io, recently joined the Automation Guild to answer the community’s common questions about Cypress.

Here are some of the top questions and myth busters regarding test automation using it.

Of course, the very first question asked was around a hot button topic Cypress vs. Selenium.

Cypress.io Vs Selenium

Any time you compare two tools, someone inevitably gets offended.

So, to set the tone, Gleb began by saying that whatever they’ve done with Cypress is because they looked up to Selenium.

Truth be told, we all use Selenium, and we have nothing but respect for Selenium WebDriver and all their other testing tools. With that out of the way, we got into a few pros of using Cypress, and some instances in which using Selenium might be a better option.

Features over Selenium

Here are three features of Cypress that do things differently than Selenium. You can decide which features matter most to your team when choosing the tools that are best for you.

1. It runs in the browser, so it allows it to interact with your code, the network, the methods, and even DOM updates in real time. Any tool that's not running in the browser cannot do that as well. So with Cypress, it's an advantage.
2. It also has built-in retries. This feature is essential because modern web applications are continually updating themselves, but they're too unpredictable. For example, during test execution, my network might take a long time, so you can’t just expect an element to appear in, let's say, 100 milliseconds. That’s why separate commands always have built-in retry and polling to ensure that even if an element takes a little bit longer, the test will pass, is not flaky and doesn't need hardcoded delays.
3. Cypress also concentrates on easy installation. With Cypress NPM, you can simply install a single command and get all the screenshots and video by default on every CI, on every platform. Ease of installation is definitely there.

Selenium Advantages

On the other hand, Selenium has a couple of advantages as well:

1. With Cypress, you have to write your tests with JavaScript. With Selenium, however, you can write your tests in any language. If you already have back-end operations with a different language, it makes sense to use Selenium because it's run on your operating system; you can efficiently work with backend operates
2. Selenium supports cross-browser testing. With Cypress, it's just on the road map, but Selenium already has it. So both have advantages and disadvantages, and you should evaluate both systems and decide what works for you.

Currently, Cypress.io Browser Support is for Chrome browsers only.

Another topic some folks had questions about was the difference between how Selenium and Cypress interact with the browser under test.

Browser Interactions

One concern that came up was how Cypress interacts with a browser, and whether it’s really driving the browser as a real user would. That’s something Selenium would find that Cypress wouldn’t.

For example, is there a difference between telling the browser I clicked on something versus emulating a real user clocking through a browser?

Gleb mentioned that at present, Cypress is a complete JavaScript solution. That means it does use a synthetic event.

It creates an event object in JavaScript and sends or dispatches it to a button, for example. So, indeed, this might not necessarily work the same way as native events that Selenium uses.

But Cypress does have native events on its road map and will have support for them soon. That will enable you to do things like file upload, download or copy/paste onto your clipboard. You’ll still have the synthetic events, but you will also have native events for all interactions.

That update, when it happens, should remove any concerns you have about the browser interactions not behaving the same as Selenium.

Mocking

When creating automated tests, you sometimes need a way to mimic the behavior of a real service. This can be achieved using a mock object.

Someone asked if it was possible to use something like the Nock mocking server for Node.js. While you can’t use Nock, there is no need to worry.

Cypress already comes loaded with SinonJs library. SinonJs allows you to create standalone test spies, stubs and mocks in JavaScript.

That means you can mock anything in JavaScript and Cypress already includes it.

Can You Use BDD with Cypress?

Another approach some teams have adopted is behavior- driven development. Can you achieve a BDD approach using Cypress?

While Cypress doesn’t have an official BDD built into its out-of-the-box option, there is a community-created BDD Cucumber processer that can help.

The NPM Cypress-Cucumber-Preprocessor plugin allows you to write your tests in BDD Cucumber Syntax, and it will then automatically translate them into Cypress.

There’s also a pretty good working sample you can check out on Github.

Cypress for Visual Validation?

You may also be wondering whether Cypress can make full-screen image comparisons against baselines in a way similar to what the Galen framework does with strict layout design requirements.

Cypress does have a built-in screenshot feature, so you can take a screenshot of a full page, a particular element or the whole thing, including the TestRunner UI. They recommend you take screenshots of a specific element so that if it acted differently, you would know which part of the page was different and wouldn’t compare, for example, a dynamic part all the time.

Worthy of note is that there is no built-in image diff in Cypress. They give you a screenshot. But it's up to you to compare.

Luckily, some folks have written plugins that actually do that.

For example, companies like Percy.io and Applitools both have plugins for Cypress that do excellent image diffing.

Cypress.io API Testing

Another common requirement for end-to-end testing is the ability to interact with an API. Fortunately, Cypress supports this as well.

When asked if REST testing was supported, Gleb recommended that folks check out cypress-example-API-testing on GitHub, which contains an example of using the E2E runner to also test Rest and other APIs.

Those are the top questions the Guild had for Gleb. What are yours? Let me know.

Transcript

A few folks mentioned that they had a hard time understanding the audio in my interview with Gleb. So here is the full transcript so that you can read while you listen.

Joe [00:00:01] Hi Gleb. Welcome to the Guild.

Gleb [00:00:02] Hi. How are you?

Joe [00:00:04] Great. It's awesome to have you. This seems to be a lot of excitement behind Cypress and also a lot of misunderstandings of what exactly it is and how it compares to other tools, so why don't we jump right into it.

Joe [00:00:19] First question “According to you, what are the three features of Cypress over Selenium?”

Gleb [00:00:29] I just want to start with saying that whatever we've done with Cyprus is because we looked at Selenium and we all use Selenium, and we all have nothing but respect for Selenium WebDriver and all other testing tools. So I will say that the three feature of Cypress that does things differently is that 1) It runs in the browser. So it allows it to interact with your code, the network, the methods even DOM updates in real time. Any tool that's not running in the browser cannot do that as well. So with Cypress, it's an advantage. 2) We also have built-in retries we thought that for modern web modern applications are continually updating themselves but they're too unpredictable. My network might take a long time so you can not just expect an element to appear let's say in 100 milliseconds so separate commands always have built and retry and polling to make sure that even if element take a little bit longer, the test will pass, is not flaky and doesn't need hardcoded delays. 3) And also we do concentrate on easy installation with Cypress npm install a single command nothing else to install, and you get all the screenshots, all the video by default on every CI on every platform. So ease of installation is there.

Gleb [00:01:59] On the other hand, Selenium has a couple of advantages too. For example, with Cypress, you have to write your test with JavaScript with Selenium, you can write your tests any language. If you already have it backend operations with a different language, it makes sense to use Selenium because it's run on your operating system so you can easily work with backend operates and also Selenium supports cross-browser testing, and with Cypress, it's just on the road map, but Selenium already has it. So you have advantages and disadvantages, and you should evaluate both systems and see what works for you.

Joe [00:02:42] It's a great answer. A lot of times whenever you talk about a tool other than Selenium so many people think you're talking bad about Selenium. We are not talking bad about Selenium; this is just an option. More options, the better every team is different, and one tool may be better for your organization than another. Not, because the tool is superior, but that's just how you to your team work better with a particularly tool. OK, we got that out of the way :).

Joe [00:03:04] Next question is, let's go to it. It's from Harry. It's been a big topic in this conference so far AI solutions — so can Cyprus currently or is it planning to utilize any AI machine learning to help prevent flakey tests? With the number of AI tools now in the market. It's a must for us.

Gleb [00:03:20] I think you can kind of understand the question in many possible ways.

Gleb [00:03:26] I kind of mentioned the built-in retries that prevent flake because of the unpredictable nature of network and storage. So there is that, and it doesn't require any ML/artificial intelligence. If you're talking about updating selectors and which selectors should be used when you're writing end-to-end tests we have the best practices guide you can use specific data attribute, so you don't depend on the Dom structure or the classes and at the same time if you want to go full way to ML helping you maintain your tests and update your selectors we are talking to companies like Mabl. Maybe something will come out of it. But we're also evaluating statistical methods that take the history of your tests and retry a failing test automatically, so a single failing test out of a thousand doesn't stop the build, but still doesn't get lost when its retried and you can go back and see which tests sometimes fail and have to be retried. So we'll evaluate every possible tool to make your test less flaky.

Joe [00:04:38] Next questions from Nathaniel. He wants to know Because Cyprus is interacting with the web app directly. Is there a layer of code, example risk above that, that Selenium would test, but Cypress is skipping? For example, would there be a difference between telling the app that I clicked versus emulating a real user clicking through the browser?

Gleb [00:05:00] Yes. So right now, Cyprus is a completely JavaScript solution. That means a synthetic event. It's creating an event object in JavaScript and sending it or dispatching it to a button, for example which might not necessarily work the same way as native events, We are adding native events. So the next thing for Cypress on our road map is actually supporting Native events. And this will enable you to do things like file upload or download or copy paste into your clipboard. You will still have the synthetic events, but you also will have native events for all events. So this will remove like tiny gaps like that. At the same time, we go back to what I do from with app action. The point of app action is not to stop testing the UI and stop clicking, it's also when your testing a feature X you go through a page, and you click on a particular button when you are testing X. But when you are testing feature Y, you don't go for a feature X again from a page. Instead, you go directly set up your space for your test then test feature Y. To avoid this duplication but flow part of interacting with the page. So you still cover all the features of the page at least once you don't repeat yourself unnecessarily.

Joe [00:06:26] My next question is, can we use Cypress.io with a BDD tool like Cucumber?

Gleb [00:06:34] So the syntax of Cypress is very different. It's declarative. It takes getting used to. But people wrote BDD cucumber to cypress tree processor which you use as a plug in to Cypress test and it that case you can write your test in BDD Cucumber Syntax, but they'll be automatically translated into Cyrpess. It's not our official plugin its something that came from the community.

Joe [00:07:04] Can Cypress do full-screen image comparisons against baselines in a similar way that the Galin framework does with strict layout design requirements. It would be a massive help if these tests would run directly from dev code, not a separate test.

Gleb [00:07:20] It's absolutely a wonderful question, and visual diffing is a great technique controller of the page looks the same. OK, so Cyprus takes screenshots it's a built-in feature. You can take a screenshot of a full page or a particular element or the whole thing including the test runner UI we actually recommend you take screenshots of a specific element so that if it acted differently you know which part of a page is different and don't compare for example dynamic part all the time. But there is no built-in image diff in Cypress. We give you a screenshot. But it's up to you to compare. Luckily people have written plugin that actually do that. Two companies that do that Percy.io and Applitools have their own Plugin for Cyprus that do great image diffing. We don't have to do anything we provide a platform.

Joe [00:08:34] What is the suggested migration strategies for page objects to app action migration.

Gleb [00:08:40] I would pick the test that takes the longest because of the painful test. It's going for a lot of action, and they all take a long time it might make sense to be mored into app action. For example, if you are logging in thru the UI and that takes a long time. Maybe you can log in by calling the app directly with username and password, and it may make an API call to shorten that wait. So my migration strategy would be to find a pain point usually an extended running test and see if some parts of it are better to be a single path app action.

Joe [00:09:23] Next question is from Christina. Christina wants to know, “By using app actions, you are not touching the UI. How are we sure that the functional elements work for example, they can actually click on an element or type in a field. Isn't this proven to error?” I think you touched on this a little bit already.

Gleb [00:09:41] You still touch the UI, and you go through a page like a user if you were testing that feature. But when I was doing a testing routing, and I needed a couple of to do it I didn't add to do it again for a bit. Instead, I default app action, and I immediately went through the depth, and without it, you still cover everything you don't repeat yourself. Instead, you use app actions to avoid going through a page again and again, so you feel will cover everything.

Joe [00:10:13] Next question is from Harrison Lee. Harrison wants to know “How do I use Cypress to test some web app in different test environments like dev/QA/production with different base URLs.”

Gleb [00:10:30] Excellent question. So in Cyprus where a lot of config value like base URL and you can set that config in Cypress JSON configuration file or provide it as a command line argument or an environment variable or even programmatically you can change it so you can point at a different base URL by just specifying on command line or setting environment variable and that way you can run the same tests against Localhost or against staging or even against production. We have a lot of examples of that in our documentation. If you search for that, you'll find it.

Joe [00:11:17] Since Cypress is on top of JavaScript Moca can it integrate well with WebDriver io instead of migrating. Also, can we use node API test in NOCK mock framework in Cyprus?

Gleb [00:11:32] Cyprus does include the Moca JavaScript engine but just to help control the before an after hooks. But the architecture of Cyprus is very different you can not integrate Moca with WebDriver.io it's more integrated with Cyprus. So, unfortunately, the first part is no.

Gleb [00:11:56] But Cyprus already included a Simon.js library. That means you can mock anything in JavaScript. It also provides network control so you can Mock and control anything that your application is sending to a server, so it's already included.

Joe [00:12:33] Can Cypress.io be configured to run with different browsers to run a test in parallel?

[00:12:38] We have a built-in , but it's not paralyzed on your machine. It's not like a single Cypress instance when you're running locally on multiple browsers it's not going to work. First of all your computer will spin out of control because Cypress is driving your app really really quickly for the test to finish so rapidly your MacBook Pro or whatever you're using will actually just become very hot from CPU usage. Instead, we allow you to parallelize your test and run multiple tests on multiple machines. So if you have a CI run, we have a service that will load balance all the tests and all the tests will be executed on multiple machines in parallel. But it's a different tool from running various browsers locally so you can follow our documentation and if the Paid, unfortunately.

Joe [00:13:35] It's great that this allows developers adding test to their code. But as a QA automation engineer, the only access we have is to the pages and not the application source code. Do you agree that end-to-end tests with pages and Page objects still has a place even with this tp technique?

Gleb [00:13:57] Absolutely. I understand. I understand perfectly. Sometimes you don't have access to the code. You cannot re-write. You cannot add test selectors. You cannot expose the object that you would like in your tests. So absolutely it's going to be for a long, long time( maybe forever) place for full end-to-end tests. You literally click on the button selected by text, by captioned, by a label and where you cannot take shortcuts, and you still have to create some kind of helper method so that your tests are reusable, easier to maintain and so on. And in Cypress its just javascript. If you like page object model, you can do that if you just like creating more utility functions that you can call, you can do that. We even have a custom command where you can write additional Cypress commands but also kind of do more stuff prove a page as you want. So our tool is flexible, and unfortunately, sometimes we have to go through all a central home page just because we cannot change the application code on that.

Joe [00:15:09] Next one is from Staz he wants to do. Do we have to pay for the Cypress dashboard service manage 500 test recordings to have test paralyzed and CI integration in Cyprus or can I get away with the free Cyprus testing or in this case.

Gleb [00:15:30] Test parallelization is a paid feature. That's how Cyprus actually makes money. The test runner is opensource, and we give it away for free. It's not future limited. Half of our small in person company is just dedicated to working on the free test runner. For us to actually give that away for free and keep development going we have to make money, and we make it on private projects, and parallelization is one of the features we can charge for. But soon if you have a genuinely opensource project that means non-profit open for library or web application or framework. We are going to release an opensource license where you can get all the feature for Cypress dashboard for free.

Joe [00:16:20] My next question is do you have a Java binding for Cypress.

Gleb [00:16:24] Unfortunately not. Ultimately all your test code has to run in a browser. If Java can be transferred to JavaScript and unfortunately it cannot, then it cannot run in a browser. We. We apologize for that. We understand, but javascript is not the language with everyone uses — even though it should be. But we've done a lot of work to make sure with our documentation, and the learning process and the errors that we get if you do something wrong are so helpful to non-javascript developers. But even if you a back end developer if you were just programming Java you still can still write effective end to end for the browser.

Joe [00:17:18] Can Cypress do remote execution like selenium grid.

Gleb [00:17:22] Not yet. So when you run in CI, you can paralyze and load balance multiple machines but each one will still be in its own browser right there in the container and will control it. You cannot control remote browser yet. We are in talks with all the companies that provide like device farm or browser. So stay tuned is just our architecture where we have to close the control and launch the browser is not. Conducive to doing that just yet out of the box and that will require technical work.

Joe [00:17:59] Next question is is the concept of app action something native to JavaScript or Cyprus. Or can it be achieved in just Java?

Gleb [00:18:09] It's native to JavaScript. It can only be Javascript because it's running in the browser and the browser only supports javascript. If you want to call that code directly, your code has to be javascript as well. Now WebDriver has you know evaluate javascript where you can kind of simulate, you can kind of send the command that will load your javascript, but it's not going to be the same. It's always going to be serialized HTTP POST request going through your browser, and then the browser is evaluating them but who knows if the object is ready actually to receive that. When you run your test and browser in Cyprus. Your javascript is executing in the same loop in the same event engine as the application so. So everything is so much easier from JavaScript.

Joe [00:19:06] Any multiple browser support?

Gleb [00:19:10] We're doing native events which are a prerequisite for multiple browsers. So in the next two quarters, I would say.

Joe [00:19:19] What browsers does Cypress support as of now? Is it just Chrome.

Gleb [00:19:23] Currently its just Chrome

Joe [00:19:45] Well Cypress work with older ASP dot net MVC applications?

Gleb [00:19:51] anything that actually loads in a browser you can test it. As long as it produces something your chrome browser can load. It's going to work.

Joe [00:19:58] Is it possible to test windows native desktop apps with Cyprus by using a combination with win app driver.

Gleb [00:20:04] Now it's very different architecture. We focus only on testing web apps.

Joe [00:20:09] What kind of reporting tools does Cypress separate support.

Gleb [00:20:12] So you can generate xunit, junit. Mock awesome. JSON report anything that Moca can create– we can create.