Mastering Selenium WebDriver [PODCAST]

Welcome to Episode 81 of TestTalks. In this episode, we’ll discuss the Mastering Selenium WebDriver book with author Mark Collin and discover how to take our Selenium testing efforts to a higher level.

Some things in life are easy to become familiar with but hard to master, and I believe Selenium one of those things. Many people I talk to say that they know Selenium, but when I start asking them specific questions, it turns out that they only have a superficial understanding. Mastering Selenium WebDriverwill take your Selenium expertise to the next level.

In this episode Mark covers advance Selenium topics like CI, Maven, BDD and much more. Check it out!
Listen to the Audio

In this episode, you'll discover:

• The #1 cause for Selenium test flakiness
• Why you need to use Maven or Gradle
• The quickest way to easily manage Selenium drivers
• When you should run your entire Selenium test suite
• How to use Docker for your testing efforts
• Much, much more!

[tweet_box design=”box_2″] The main causes for automation flakiness is badly written #Selenium tests ~@Ardesco [/tweet_box]

Join the Conversation

My favorite part of doing these podcasts is participating in the conversations they provoke. Each week, I pull out one question that I like to get your thoughts on.

This week, it is this:

Question: What do you think are some of the main causes of test automation flakiness? Share your answer in the comments below.

Want to Test Talk?

If you have a question, comment, thought or concern, you can do so by clicking here. I'd love to hear from you.

How to Get Promoted on the Show and Increase your Kama

Subscribe to the show in iTunes and give us a rating and review. Make sure you put your real name and website in the text of the review itself. We will definitely mention you on this show.

We are also on Stitcher.com so if you prefer Stitcher, please subscribe there.
Read the Full Transcript

Joe:         Hey, Mark. Welcome to Test Talks.

Mark:       Hi.

Joe:         It's great to have you on the show. Today I'd like to talk to you about your awesome new book, Mastering Selenium WebDriver. Before we do, can you tell us just a little bit more about yourself?

Mark:       Well, I'm a tester, basically. I've been doing this for about fifteen years now. I got into automation about six years ago and that's when I got into WebDriver. That's about it, really. I guess I've just carried on from there.

Joe:         Awesome. How did you get into actually programming and automation six years ago? Was it just part of the job and you just stumbled into it or is it something you always wanted to do?

Mark:       I've always liked fiddling with code. To be perfectly honest, I'm slightly lazy. I don't like doing the same repetitive task again, and again, and again. Automation seemed like an ideal way to be lazy and to also fiddle about with code and tweak with things.

Joe:         Why did you write Mastering Selenium WebDriver?

Mark:       I originally got asked by Packt Publishing if I'd like to do a second edition of an existing book. I didn't really like that idea because if you do a second edition you start cutting into the original offer's royalties, which seems a bit off, really. I wouldn't really want somebody taking my book and adding an extra paragraph. You know, taking away from it, as it were. I said I wasn't interested in that and then they suggested I could do my own book and that's what happened, really.

Joe:         This book covers lot of more advanced topics. I know there's other books out there for Selenium, but there's things in here that you covered that I don't see in other books like Maven. Things that may seem simple to someone that's done automation and programming before, but for someone that's kind of new to automation, I think concepts like Maven can sometimes cause them problems. Could you just tell us a little bit more about the Maven piece? Why is Maven important and when would you use it?

Mark:       Well, in the UK anyway, I've worked in London quite a long time and I've worked with quite a lot of companies in the UK. Maven is bread and butter in the driver development community over here. In all the enterprise software you work on, you'll normally see Maven being used as the [building dependency 00:02:12] management system. There doesn't really seem to be that much out there talking about Maven and using Selenium with Maven. Most of the information out there is very basic or it doesn't really talk about how you would deal with dependency management or how you'd build things. It seemed like a good opportunity to put that in because I haven't seen it around before.

Joe:         That's why I was really excited to see it. Like myself, I use Maven with Java, and I always get weird issues like Maven dependency issues and plug-in issues and it's not always as straight-forward as you would think. In your book you actually start a project from scratch and you show all the different palm file entries you need to use in Maven in order to get it working. I found it really helpful.

Mark:       It's basically something I've been doing for quite a long time. I've slowly built up a very basic template of getting everything together so that you can just start a new job and dump in a very basic framework quite quickly. It just seemed like a good thing to put in, really. It seems useful, and if you step through it slowly and tell people how it's done, hopefully they'll be able to understand why certain things are done certain ways and it will be useful for them, really. It's just a good start for everyone.

Joe:         What would you say the benefit of Maven is? Right now with people using a tool like Maven, they have to manually resolve all the dependencies every time there's an update. With Maven you just update it and it takes care of all the things in the background for you automatically?

Mark:       Well, the main idea of Maven is you can just check out a project, type Maven clean install, and be ready to go. It will download all the dependencies, it'll set up your class path, it'll do everything for you. It will create artifacts. It will create your jars or your walls, depending on what sort of output you're getting from your build. It'll also run all your tests so you'll build an entire project, run one of your unit tests, it'll create an artifact that can be deployed to a container. You can then deploy it to a container as part of that build and you can run all of your Selenium tests against it. You can be pretty sure that you've got a working bit of software that's well-tested and you don't really need to do anything once it's all configured correctly.

Joe:         Awesome, and what are the other benefits? I think you've actually written some plug-ins, so what is the driver binary downloader Maven plug-in?

Mark:       That's a little plug-in that I created to get around a problem that most people have when they're using Selenium. To get Chrome driver, IE driver, and [inaudible 00:04:46] us much anymore, working, you need to download and executable. This executable is basically the layer that talks to the browser. It talks between the browser and the Selenium bindings. When you write your Selenium tests, they need to talk to a bit of code that can then talk to the browser and tell it how to do what you're telling it to do to automate the browser, basically. These executablea aren't held in Maven because it's not Java fast, basically. It's a standard executable for your Windows or for OS X or for Linux.

Because when you do a Maven build you want everything to just download and work by itself without having to do an manual configuration, I have wrote a little plug-in that will also actually download these esecutables and set up the environment variables so that it just all work. The idea is that if you're using that plug-in with a Selenium framework, you can type in Maven clean install and it will pull everything down for you and run your tests and you don't need to do the manual configuration on your machine or set up any environment variables in your local machine.

Joe:         Awesome. I was really happy to see this plug-in also. I haven't tried it yet, but I'm definitely going to. The reason why is I support eight to ten Scrum teams, and a lot of times these teams, when they're starting with the framework, they don't remember to pull down the IE driver or they don't have the environment variable and I have to trouble-shoot twenty different configurations. Thank you for writing it. Having it's a big help, so that's cool. Another piece is that with Maven, do you know … I've been hearing more and more about Gradle. If someone was starting from scratch, would you still recommend they use Maven? Or is it project by project basis whether or not to use something like Gradle over Maven?

Mark:       Completely project by project basis. I would recommend you use whatever your team uses. If you've got a lot of expertise in your team using Maven, it makes sense to use Maven because if you get stuck you'll like to have someone there who will understand it to help you out. Exactly the same for Gradle. If you've got a team that's got a lot of people who've used Gradle and know it very well, then why not use Gradle? You know, there's nothing to stop you. I don't see these things as there is one way that you should do it and you shouldn't touch anything else. It's all about using the expertise in your team and doing something that's sensible. If you don't know Gradle at all, it doesn't really make sense to try and use Gradle in your projects unless you're trying to learn it. Then I suppose it depends on what you're doing. I do quite a bit of consultancy work and generally speaking, the clients don't want to pay me to learn something I don't know, so I'd probably go for something that's tried and tested. If you are in a team where you are all trying to learn something and you want to give Gradle a try, why not?

Joe:         Awesome. Yeah, I think it's a great answer. If your team's using it, why not use it? If they're not, why add another extra layer of something that you're going to have to teach everyone else about? I think that's a great, great answer. With Maven, do you know, can you pass custom parameters? For example, I know I've seen teams that actually create batch files that do this, but I think there's got to be some functionality within Maven that allows you to do a lot of things that people aren't aware of. Are there anything about Maven that you think a lot of people don't know you can do that they do a lot of hacks for?

Mark:       Well, Maven's quite powerful. You can pass in variables in the command line when you start Maven up, so if you do like a minus D variable name, you can set a value for a variable that's passed into the Maven build process. In the work I think I'm using passing in Maven command and variables to set which sort of driver you want to use so you can switch your driver from Firefox to Chrome driver or IE driver using a command line attribute. Maven also supports profiles which are really useful. You can profile away your Selenium tests so that by default your Selenium test won't run on a standard Maven clean install, but if you add a profile like a minus P Selenium tests, if you've hidden your Selenium test behind a Selenium test profile, when you specify that profile it will run the Selenium test as well.

That gives you a way to give your build different abilities depending on where it's running. If you're running your build in your local machine you might want to run everything, whereas if you're running from the build server you might want to cut down the build and only run certain parts at a time. You might want you break your initial build with a unit test as the first step, and then the second part of your build where you build the integration tests or the Selenium tests, is a separate phase.

Joe:         Awesome. That's actually once again functionality I wasn't aware of, and I think that's really great. One of the questions I was going to ask you is in your experience do you usually recommend that developers run the whole test suite, or do you do something like you create a profile and you only run maybe half the tests? What would you recommend for that particular process to developers? Do they run the whole entire test suite or do you usually just have them run particular features that you know are being affected within this particular sprint that they might be working in?

Mark:       I think they should always run the whole thing.

Joe:         Oh.

Mark:       Continuous integration is there to make sure that things work, but you shouldn't be checking code that you're not sure if it works or not, really. I mean, I see continuous integration as a safety layer. If it works in your machine, you then put it on to the CI box to see if it works on your CI environment. Your CI environment should be similar to your production environment, really. I use a Mac to do my work, so if I can build my stuff on a Mac, that's great, but if the code does [inaudible 00:10:23] building and testing and my local Mac is actually part of the Linux box, that's where you'd have a CI server that's running in the next. That would just tell you if there's any weird things that are going on that work on OS X but doesn't quite work on Linux.

Also for Windows. If you're using Windows and use [inaudible 00:10:39] it might work fine in Windows but when you put it into a unit spaced environment, there might be a couple of quirks that you're not quite sure about. I see that as the point to CI. It's a safety layer. If you're constantly breaking a build I think you're probably not doing it right. We don't want builds that are always broken. We always want working code because broken builds slow people down. If I break the build, somebody's got to go and find out why the build's broken. They shouldn't be at all committing on that build because if we commit on a broken build we don't know if we're breaking something else. It's bad practice, really, to not run your tests. I do see people not running all of their tests all the time and that's normally because they've written so many tests that the build process takes a very, very long time.

You'll see some legacy apps where people have done that a lot and then [inaudible 00:11:28] testing, and the build process will take maybe two hours for them to actually run all their tests. That's a bad thing. You shouldn't have a build process that's two hours long. If you've got a build process like that, you need to have a look at what you're doing because you've probably got far too many [inaudible 00:11:42] tests. It's probably worth getting rid of some of those antiquated tests that aren't useful and moving them down into the unit testing layer.

Joe:         That's a great point, and that's actually something I've been struggling with for a while. Like I said, I work for a large company, and for some reason we used behavior driven development, but they have to have a feature file and a scenario for each requirement so they have each one as a requirement. We have thousands of tests and it takes eight to nine hours to run. It's not really sustainable, like you said, because people can't run it on locally. It takes forever and no one knows how to build up these environments and it's just a nightmare. When I try to tell them, “Hey, let's do more unit tests, more integration tests,” it's always a struggle to say, “Well, no. That's not how are users are going to use the application.” I know you have a whole chapter within your book on creating faster feedback loops. Do you have any tips on how to get around this particular issue or issues like this?

Mark:       I guess really showing people various talks by people [in the industry 00:12:44]. JB [Renspocher 00:12:46] does a brilliant talk about why integration testing is a scam. He basically talks through what you're trying to do if you're doing integration tests, which is basically you're taking the outside layer of a program and you're trying to exercise every internal path through the outside layer. Now to do that from the outside layer is very, very hard. You've got to come up with hundreds and hundreds of different scenarios to try and hit all of these different paths in your code, and you're probably not going to manage to do it.

If you can move those scenarios inside the actual code and do them as unit tests, you can do a lot of testing on specific functions and hit all of these different paths. You don't need as many tests to do it because you're breaking them down into smaller easily manageable chunks. I think that's the way to go, really. I mean, there's always going to be a case to have some [inaudible 00:13:36] tests, but really [inaudible 00:13:38] tests should be checking that a major functional path works. A good example I see quite a lot is when people are actually in tests they spend a lot of time checking text on the page. I don't think you should ever do that, really. You should have a test that checks that however you're putting your text on the page works, but once we know that mechanism works, there's really not a lot of point in checking what the actual text is, because the text is always going to change. It's one of those things that makes for test [inaudible 00:14:07] really a big waste of time.

Joe:         I definitely agree with you. I see this all the time. We have a rest service where it generates information that's going to be generated in the front end and people still use the UI to test when they could do a majority of the test against the rest API and get the coverage they need. Yet they still go against the UI, so it's a constant battle.

Mark:       That's a good point because if you've got a rest API that's giving you all the text for your front end, it's probably all coming out as a big blob adjacent. It's very easy to do a comparison, right, for one big blob adjacent and see if all adjacent you're getting is the adjacents expected. Then if you take that adjacent and you pump it into a front end and you use front end tests to try and test each line of text that is in the page … You've not just got text that's apparent on the page. You've also got things like old text for images, pop-up text, titles, all sorts of things. You might even have text for sight readers. A lot of that text you're not going to see on the front very easily. You can get to it, but it's awkward and it takes a lot longer to try and do that sort of testing from the front end. It really isn't a good idea and a lot of people don't seem to realize this.

Joe:         Mark, also you talk about living documentation. Your test has living documentation. What do you mean by that?

Mark:       I got fed up with people talking about regression tests. The idea of a regression test is to see what's changed. If we've made some code changes to our program, we know it's changed so we don't really need a regression test to tell us that it's changed. When we make our code change we're obviously going to change the tests that we already have in the system so that they mirror this code change. Really what our tests are doing is it's documenting the behavior of the program. We're not doing a regression test. We're just showing people how our program works. Because of that, that's why I see tests more like documentation.

They're not documentation in the way that it's just a bit of text that people don't have to read, it's living documentation that's constantly changing. As your application changes, your documentation changes with it. I guess that's what I mean by test as documentation. It's telling you that your program always works in a certain way. It also means that if you make some changes locally and you break some of your tests, it's probably not that your documentation is wrong so you have to change those test, which is in effect changing your documentation, or you can revert your functionality back to the way it should be working, i.e. it matches your documentation.

Joe:         Cool. I think a lot of people, for some reason, when they hear living documentation, they automatically assume, “Oh, we need to use behavior driven development.” When they do that they end up using BDD as an automation framework rather than as a collaboration tool, and so they add an extra layer when they could have just created a test in a more readable way and have the benefit of readability that way rather than have an extra layer like BDD to create this.

Mark:       BDD, I think, it's easy as possible to add a lot of people who don't really know what it means. To me BDD is just having a conversation. That's what it means. Talk to your users, find out what they want. You don't have to use a tool like Cucumber to be doing BDD, for example. You don't need to write all of your [inaudible 00:17:26] tests in advance before you write code, which some people seem to think. BDD is just talking to your users and making sure that you're building what they want. I get quite frustrated with BDD a lot of the time because I've come up to quite a few projects where they've got hundreds of stack files with Cucumber and you say, “Okay, so which business analyst reads this? Which of your users read it?” “Oh, none of our users read it. We just use Cucumber because we want to do BDD.” It's really frustrating for me. You've completely missed the point of BDD and instead you're using a tool to try and say you're doing BDD.

Joe:         You also bring up a few points besides BDD of ways to help run tests quicker. One of them was using Docker with Selenium. Why would you use Docker or when would you use Docker, especially when, once again I have teams that say we only support officially Windows with Internet Explorer or Firefox so we would never run against Linux? It's just a browser, anyway. What would you say to someone about that? When would you use Docker or when do you think it's appropriate to use something like Docker?

Mark:       Okay, so the reason I'd use it for Selenium tests is because [inaudible 00:18:37]. If you want to set up a Selenium group really quickly, you can do it in Docker. You've not gotten to Internet Explorer, so if you do want to do some testing on Internet Explorer, you're going to have to manually set up a Internet Explorer agent like you normally would with a grid. It's just a nice, quick way to get you started, basically. You'll get yourself a grid hub and you can get as many Chrome or Firefox agents as you want attached to the grid. That's why I like Docker for that particular scenario. If we're talking about Docker as part of a text hack in more detail, I would much prefer people start creating Docker container [inaudible 00:19:19] versus their application as part of their build.

If you do that it makes it really, really easy to test because you just spit out the Docker container and then you can fire your internet test at Docker container. If a Docker container … Sorry, if all your tests pass, you know that your application that's in that Docker container is a good application that you can move down to a different test environment. Now, with Docker, because it's a container, you can just take that container and move it anywhere. You can put it into an integration testing environment, a performance testing environment, a staging environment. You can even take that same container and stick it in Live.

We're not making the code changes at this point, we're just using exactly the same container with exactly the same application that we used when we started doing our testing. We know for a fact that what we've tested is the same as what's live. [inaudible 00:20:13] we get from that is that if something goes wrong when it's live, we can just take a copy of that container and we can hand it back to the development team and say, “This container went wrong. Here's the actual code as it was running. Have a look and see what you can find wrong with it.” It makes it a lot easier to replicate problems and fix them.

Joe:         Awesome. Once again, you have a chapter in your book called Hooking Docker Into Selenium and you walk step by step how to do this. I guess the biggest question I should have asked or the higher level picture it was, for someone that doesn't know, what is Docker at a high level?

Mark:       Docker is kind of like a VM, but it's not a VM. Linux has the ability to isolate some processes inside a Linux box so that those processes think they're running in a completely isolated way that is not attached to the rest of the Linux machine. That's basically what Docker does. It hides the processes in a sand boxed area. It's incredibly quick to start up because you're not starting up an entire VM with an entire operating system. You're just starting up a single Linux process.

Joe:         Very cool. It sounds like something like rather than having a physical machine that you have to maintain over time, you can just create these on the fly, tear them up and tear them down fairly easily.

Mark:       Yeah, exactly. You do still have to have a physical machine. You've got to have a machine that runs Docker, but that means you just got to have a Docker host that you manage rather than the rest of it. That can be good, that can be bad. It depends on how much you like Docker, I guess.

Joe:         With Docker right now, it only supports Linux, but I believe in your book you mentioned that Microsoft is working with Docker to create some sort of integration?

Mark:       Yes. Next [inaudible 00:21:55] Windows server is supposed to support Docker containers. I think they're got some big stuff working at the moment but I'm a little bit out of date so I'm not sure right now what the state of that is. I mean, it does sound good going forward because if we can run Windows containers and Linux containers using regular computer support, that basically gives us our entire test stack. Okay, we won't have our OS X containers, but realistically OS X isn't usually used in the world for as a server that most people host their stuff on. Using laptops quite a bit, yeah, but not as a server.

Joe:         Mark, also in your book, you have a chapter on Exceptions Are Actual Oracles. Once again, I work with a lot of teams, and for some reason they don't read the exceptions. When they do for some reason they don't understand it. I think you outline the most common exceptions you'll encounter with Selenium. Could you just tell us a little bit more about what are exceptions as actual oracles? What you mean by that?

Mark:       I guess I mention it in the book quite a bit. I see an oracle as something that gives you information. These exceptions are telling you why something is broken or something hasn't worked. These exceptions are actually trying to tell you how to fix the problem you've got. A lot of people will see an exception, they'll see loads and loads of code, and they'll freak out because there's about fifty lines of code that they don't really recognize. They won't look at it in any great detail and they'll just ignore it. They'll start trying to change things in their code and run it again and see if it fixes it and change it again, and again, and again. They don't really know what they're doing. They're just flapping about like a fish out of water trying to hope something magically fixes itself. If you'll actually stop and take a look at the information that these exceptions are giving you, nine times out of ten it will point to an exact line of code where you've got a problem and it'll give you an exception name that means something very specific that'll help you find out what's wrong with your test.

Joe:         Are there any tips on how to make a Selenium test more reliable and less flaky? I hear every conference I go to, Selenium tests are flaky. What does that mean? What do you see as the main causes for this flakiness?

Mark:       I think the main causes for flakiness is badly written tests, which people don't like to hear, surprisingly. Selenium's very, very good at what it does. It does have a few quirks, but the vast majority of flakiness is in tests weighting problems. The weighting problems will be that the page isn't in a state that the person whose testing expected it to be in or it could be that the page is changing and it's not changed quick enough or it's changed too quickly. A lot of the time it's people don't really know the application they're testing is doing under the hood. A good example would be things like stale element reference exceptions. When you're writing your automated tests, occasionally you'll get a stale element reference exception. What this means is that you found an element [in the dominant 00:25:15] website and you're now trying to interact with that element. Since you found that element, any time you tried to interact with it the code on the web page has destroyed that element so it's not there anymore. At that point you get a stale element reference exception.

Now a lot of people will then go and have a look at the page and they'll see an element that looks exactly the same. It's in exactly the same place [in the dom 00:25:39] and it's got exactly the same mark up. [inaudible 00:25:43] Why has it gone wrong? What's actually happened is that the web page you're testing has destroyed the element and it's recreated it again. Even though it looks the same, it's a completely different element. The person that's writing their test hasn't realized that this is what's happened so after [inaudible 00:25:59]. What they really need to do is work out what the conditions are for that element to be destroyed and make sure that whenever that element is destroyed, they try and find it again, or try and find the replacement element. It's normally knowledge, I guess, really, when you first start doing Selenium. You're not really aware of time out issues or things like stale element reference exceptions. The more you work with it, the more you get used to it.

Joe:         That's actually a great point and I'm going to have a rant right now, I feel like. It's just that once again I work with eight to ten Scrum teams and they're always telling me, “Selenium doesn't work with our application.” They just do halfway scrips. They don't realize what they're doing. They don't put the effort in. So what do they do? They blame Selenium, not their coding skills, how they're creating the test or even the application. They'll be demo-ing the application. You'll be seeing these weird pop-ups and they'll be clicking okay on them. If you were to actually run an automated test against that, those pop-ups would have killed it, but they blame Selenium. They don't blame the application and what they're doing. It drives me crazy.

Mark:       It is one of the more frustrating things I see. I can't think of a time when somebody's told me it seems flaky where it hasn't been a badly written test.

Joe:         No.

Mark:       Another reason is very bad locators. Writing tests these days, there's very little reason to not get the developers to put in an ID for you to find an element or to just do it yourself. A lot of people you'll see have written these massive XPaths that are about [inaudible 00:27:29] paper, and they're insane. XPath is very powerful. The XPath was originally designed for xml documents, not for html documents. The way you write an XPath or an html document is not the way you'd write an XPath for an xml document because they're totally different things. Xml documents don't change. They've got schemers that are very static whereas an html page is constantly changing and whenever somebody does something, the front end will change its structure a bit and start to get added into the dom and stuff get taken out of the dom. The dom will get manipulated and then attributes will change. You'll get extra classes [inaudible 00:28:10]. You'll get different colors appearing all over the place. You don't want really, really long and explicit XPaths, you want very loose XPaths finding a generic thing.

Joe:         Before we go, is there one piece of actual advice you can give someone to improve the Selenium automation efforts. Let us know the best way to find or contact you.

Mark:       I guess if you want to get really good at any automation, the best thing you can do is learn how to program. I get a lot of people asking me how to get into Selenium or how to start automated testing. Really you've got to understand how the programming side of it works. Don't use things like the Selenium ID plug-in for Firefox, because that won't generate good code for you. You can't hope to write good scrips if you don't understand the basics of programming. I think the best thing to improve your automation skills is to improve your programming skills.

I'm on Twitter. My hashtag is ARDESCO, that's A-R-D-E-S-C-O