Using the Open-Source Automation Framework White [PODCAST]

Welcome to Episode 77 of TestTalks. In this episode, we’ll discuss White, the open-source framework for testing WPF and Window applications. You'll be surprised what you can automate with a free tool.

Not every tool is right for every automation effort. When most people think of open-source automation, they think Selenium. But Selenium is only for testing browser applications. What if you need an open-source solution to also automate your WPF and windows applications? No worries. In this episode, you’ll discover how to automate non-browser applications using the White framework with Vladimir Khorikov, the Pluralsight author of Automating UI Tests for WPF Applications, as he shares some of the best ways to create sweet and robust automation for WPF applications and more.

Listen to the Audio

In this episode, you'll discover:

• How the test pyramid is different for WPF applications.
• Why White might be a better option than CodedUI.
• Is the White framework right for your project?
• Tips to improve your White and WPF automation efforts
• Some best practices you can apply to any automation projects.

[tweet_box design=”box_2″]#Tests written with the #BDD framework are better than usual documentation~@vkhorikov testtalks.com/77[/tweet_box]

Join the Conversation

My favorite part of doing these podcasts is participating in the conversations they provoke. Each week, I pull out one question that I like to get your thoughts on.

This week, it is this:

Question: What do you say to a manager or co-worker who thinks BDD is just about automation? Share your answer in the comments below.

Want to Test Talk?

If you have a question, comment, thought or concern, you can do so by clicking here. I'd love to hear from you.

How to Get Promoted on the Show and Increase your Kama

Subscribe to the show in iTunes and give us a rating and review. Make sure you put your real name and website in the text of the review itself. We will definitely mention you on this show.

We are also on Stitcher.com so if you prefer Stitcher, please subscribe there.
Read the Full Transcript

Joe:         Hey, Vladimir. Welcome to TestTalks.

Vladimir:  Hey, Joe. Thanks for having me here.

Joe:         Awesome. Today I'd like to talk about your Plurasight course automating UI test for WPF applications but before we do, can you just tell us a little bit more about yourself.

Vladimir:  Sure. I've been working as a software developer for about 10 years now. I like learned and applied software developments patterns and best practices and other degrees that help you with successful software projects but when I first started in the IT industry professionally, I worked as a career person. I'm familiar with this side of the development process as well.

Joe:         Why did you create the Pluralsight course on WPF applications? Is that something you use often in your job?

Vladimir:  Yeah. Actually, I had an interesting experience in one of my previous projects. I thought it would be nice to have some kind of documentary on that so that if other developers have similar problems, they could just learn from it. My colleagues and I struggled through this experience. This course is really the essential part of it. That's actually quite interesting how I go there. When I joined the project, it had 5 years of history already. At that point, it was pretty clear it needed re-packaging, not just some kind of clean-up here and there but a re-packaging that will affect something [dimensionally 00:01:22] being as its architecture.

The cold base contained unit tests but the scale of their [inaudible 00:01:27] meant that we will need to write most of them as well. We didn't have a cover for our [inaudible 00:01:34] activity. It was pretty scary for us to do that without it. The only way out that we saw the point is to create a set of high-level tests which wouldn't know all the applications and internals. They would verify the system from the end user's point of view. It was the only type of tests that could potentially survive in the process of re-packaging. That's how I got there. My initial goal wasn't to build a test [inaudible 00:01:59] in the first place but it was to survive the new re-packaging.

Joe:         Awesome. You actually bring up a great point in your course. You talk about the test pyramid ratio is different for WPF applications instead of browser. Can you just talk a little bit about what that difference is?

Vladimir:  Yeah. That's a great question actually. Just for those people who isn't familiar with the test permit, I'd like to give a quick [inaudible 00:02:23] here. Generally, people point you 3 types of automated testing, a unit integration and UI. They form the test permit which means that the number of unit tests is larger than the number of integration tests. At the same time, the number of integration tests is larger than the number of UI tests. UI tests in the test permit are also called acceptance or functional tests. We have some kind of pyramid here, meaning that we have a large foundation in the tests and the small apex which is [inaudible 00:02:56] tests.

The classic test permit usually contains very few UI tests because it's usually the hard test the system as a whole. The application may contain a lot of separately-deployable components or in the case of web application, lots of dynamic Java script. It's really hard to make it work usually. The tests may not be as reliable as you want them to. That's why the notion of the test permit tells us to focus on the unit and integration tests primarily. However, [inaudible 00:03:28] WPF applications is different. We still need to employ unit testing, art of unit testing, but in many cases it's often easier to create automated UI tests or other data integration tests.

Moreover, UI tests provide more value because they verify the whole stack, not only parts of it. The test permit for WPF applications often consists of only 2 layers, unit tests and UI tests. Unit tests still outnumber UI tests but the amount of UI tests seems pretty large as well.

Joe:         Awesome. That's a great explanation. How did you decide what tool to use? I know there's a few options you can use for WPF applications. You could use a vendor tool like UFT or you could use CodedUI. There's also an open source tool I think called White. Can you break down how you decide what tools to use and what the differences between them?

Vladimir:  Sure. We took into account 2 major ones. One is CodedUI and White. CodedUI is a technology from Microsoft for creating UI tests. It's a [fragment 00:04:31] product in the Microsoft stack. It has a quite powerful [inaudible 00:04:34] actually. For example, with CodedUI you can write tests for both desktop and [web 00:04:41] applications. Desktop and web applications can be tested in a single test which is pretty handy sometimes. Also, CodedUI has a rich recording capabilities which may be useful in some cases.

White is an open-source library from Test Tank. Like CodedUI, it has a more narrow scope. It only targets desktop applications, not web applications. Both CodedUI and White are based on UI Automation Library for Microsoft. It's a low-level library. In theory, it is possible to use it directly for your tests but you practice, if it's much more convenient to use as a CodedUI or White because Automation Library itself is difficult to use. How we decided what to use for our project, basically I think the choice should depend on the goals you have.

If you intend to write just a couple of tests or the application you are testing doesn't change much, you are better off choosing the CodedUI because it allows you to easily record the tests. You can have you test suite without a lot of effort which is good for small projects. On the other hand, if you intend to write large-progression test suite, you should definitely choose White. There are 2 reasons for that. First of all, White is multi-format than CodedUI and not just a little bit. The difference is quite big. The difference in the tests I made, White ran 5 times faster than CodedUI in the same scenarios.

Joe:         Wow.

Vladimir:  I wouldn't say performance is the most important thing when it comes to writing UI tests but it still can be substantial. If you're automated, tests will be large. It's just better to run the tests within 15 minutes on the single-version machine that is in half an hour on 3 machines. The more important feature, however, is the [multi-ability 00:06:38] of the tests. I believe the degree to which you can easily change and extend your tests is crucial in the long run. In my experience, it is the single trait that makes the successful UI test suite. When it comes to multi-ability, it turns out that with the recorder it's pretty much impossible to create maintainable UI tests.

Recorded tests are too brittle because changing a single piece of UI causes all the tests that rely on that part of the UI to break. When that happens, we basically have 2 options. Either we record all the broken tests or try to figure out how to change the generated tests directly. The recording is not the best option, as it often requires a lot of work, so once developers end up with fixing such issues manually. CodedUI actually allows to do that. When recording a test, it places all the UI elements it encounters into the files called UI maps.

For example, if while recording I click on a save button, this button will get into UI map with some information about how to locate the control on the screen. Usually it's the caption on the control. In this example, it would be safe stream. The idea behind UI maps is good. I think the greater sub-quality why you wanted to find some kind of balance here. I think they wanted to allow the developers to easily record the tests on one hand. On the other, they also wanted to help them change the generated code, if necessary. In the reality, however, this entails some other problems.

With UI maps, it's hard to keep up with the teamwork because of merge problems. As the code of the UI maps is origin-related, it's really difficult to resolve merge conflicts. Often what ends up happening is developers start using some kind of the lock mode here. What I mean by that, when one of the developers work with a UI map, they inform other team members so that noon else can work on this map at the same time. As you can imagine, that's quite cumbersome.

Another issue with UI maps is that it requires a lot of work to make them readable because they tend to quickly get blotted. That's elements. Then it becomes really hard to understand which element stands for what. Overall, I would say that although the recording capabilities of CodedUI look nice, it's hard to make use of them in large projects. If you want to be able to maintainable tests with your [inaudible 00:09:08], do it manually and not just delegate the tasks for recording.

Joe:         I completely agree. I've been doing test automation for 15 years. Any time I see one just doing a record and playback, that's usually going to make it really hard to maintain going forward. I definitely agree with you.

Vladimir:  I wish we'd figure out this earlier.

Joe:         With White, what technologies can it handle? Is it just for WPF or can you use it also for standard Windows applications?

Vladimir:  Yeah, you can actually. Because it's based on UI Automation Library from Microsoft, it allows you to handle all kind of desktop applications that works for Windows. It can do both, WPF applications and also native Windows applications.

Joe:         Awesome. If someone wants to learn or get started using WPF, what do they need? Since WPF is open source, I guess they would need Microsoft Visual Studio. Do they need a certain version of Virtual Studio?

Vladimir:  Yeah. You definitely will need Virtual Studio because I'm not sure if there's any other options for building that web applications because it's quite closed. I would say it's not as open platform as, for example, [inaudible 00:10:22] NBC. Yeah, you will need Visual Studio for that. As far as version goes, I would say that it is available from the very beginning. I think it's available starting from 2010. You can even use it with 2008 with some service pack I guess.

Joe:         In your experience, how reliable have your White automation tests been?

Vladimir:  That's a good question. I have some experience with Selenium and [inaudible 00:10:54], not much, but I have some understanding of how the automation looks like for web applications. If we compare this process to other process of automating applications for Windows, what I see here is that desktop software automation works more reliably overall. I think that's related to the nature of HTML and Java script. There are much more little things that could go wrong with web applications than it is with desktop apps. UI issues, UI bugs, in desktop software are more reproducible which gives the tests more stability.

Both CodedUI and White I think do a pretty good job here. If a test fails, in most cases you can be sure that there is something wrong either with the test itself or with the application. I personally never ran into a situation where a test written with White failed but passed after I ran it again. I think it's pretty stable and reliable.

Joe:         Awesome. No, that's good to know. Can you just walk us through how you develop your automation scripts and what the lifecycle looks like.

Vladimir:  First of all, we have our requirements from our analysts. Then our test writers write tests for them. Then we develop [inaudible 00:12:11]. After that, after all the functionality is tested and all the bugs are fixed, we then write our UI automation tests to cover this activity, so that testers won't have to test them again when they do their [regression 00:12:30] tests. Basically, with our automated UI tests, we reduce the regression test like the UTF for the [inaudible 00:12:38] people so that they wouldn't have to run them over an hour again with every release.

Joe:         Do you have developers that handle that activity or do you actually have some people that are test automators on the team that handle that activity?

Vladimir:  No, we don't have [inaudible 00:12:55] persons. We have only K persons that test our software manually and developers that develop it but we had some dispute about who should write automated tests. The dispute is there because there is some delegating this activity. On one hand, K people are in charge of testing. On the other hand, developers are the ones who automate things. One could think that it's a good decision to give this kind of work to K department but in my experience in my project, this didn't work out.

The reason for that I guess is that the creation of UI tests requires programmer experience. There is much more of automation in this activity than it is of manual testing. Actually, in the last project we had this exact situation. The automation activity was given away to a K person who was a smart guy but he just hadn't enough programming experience to work on this kind of a task. We got the first few tests pretty quickly but after that, the process of creating new tests slowed down. The existing tests often broke. It took him a lot of time to keep up with them. [Inaudible 00:14:07] got worse and worse over time.

We eventually discarded all the tests and started everything from scratch. This was our first attempt to automate our solution. We didn't succeed with that. After that point, we had to completely rethink the process of creating automated tests and involve software developers, me included, to test automation.

Joe:         I completely agree. If you have a QA resources and you want to create a really robust framework, you really are developing an application to test other applications. I definitely agree with your approach of having your developers send a lab piece and have the QA people focus on what they do best. That's the actual testing.

Vladimir:  That's a good point actually, so the applications that test and not application. I like it.

Joe:         How does this work? Does your test run into this integration environment every night or your regression test may only run after the end of a sprint? I'm just curious to know when these tests run and how often.

Vladimir:  In our case, we run it at every check-in actually. This library is just a library. You can use it when you have any test run or continuous integration environment of your choice. We use [T Fast 00:15:18] for that process initially but then we integrated to [T City 00:16:22]. The tests works in both of them quite nice. It was run on every check-in. The only caveat here I guess is that you should keep your virtual machines unlocked. Otherwise, Windows wouldn't allow the test funnel to interact with the UI. It might be cumbersome in some cases but it's doable.

Joe:         Cool. In your experience, are there some common things you think that most people starting off with WPF automation using White may encounter?

Vladimir:  Yeah. I would say that if you have chosen White for a large project, then you are on the right track already. The White library has a really nice object model which how to write good-looking code for your tests. It's pretty easy to follow the [inaudible 00:16:09] and best practice with it. I mean such practice as SAS encapsulation, [inaudible 00:16:13] of concerns and so on. White doesn't hinder writing good codes, so I would say that the common difficulties here is not the White library itself but the common difficulty is to actually write good code.

That's a separate, big topic which involves many program and best practices. I would say that the most important one here is to keep the code of your tests as simple as possible. I would even say that simplicity is the single most important attribute of your tests. Not only should you be able to easily recognize the meaning of your tests but it should be obvious for your colleagues as well.

Joe:         No, that's a great point. I t may seem obvious but readability is one of the biggest things for any sort of project I've noticed, making it maintainable. That's a great point. Do you use SpecFlow for behavior [determine 00:17:04] development?

Vladimir:  Yes. I talk about it in the last module of the course. Basically, the idea is that you can transform all the UI tests you have into human readable text using SpecFlow. This actually unlocks many good benefits for the project. This enables you to create some sort of [read 00:17:27] documentation which is useful for all team members, not only K people or software developers. For example, it can be also used by analysts or project managers. Tests written with the BDD framework are better than usual documentation in 2 ways actually.

First of all, it never gets outdated because the tests are executed constantly. If any of the functionality that they cover gets out of sync, the tests will notify you about that. Secondly, such tests are a good candidate for a single source of truth. In a large project, the documentation is often broken into several places. It's spread across different sources. Sometimes those sources even compete with each other. It's hard to compile the single recommendation out of them. UI tests written with a BDD framework such as SpecFlow is a good replacement for them because they never lie abou the current state of the system.

Joe:         That's a great point. It's almost like living documentation. You're not creating this documentation up-front. It's growing with it. Talk about once you run your tests and say they do fail, are there any tips you have for creating reports, so for debugging that makes it easier to know that when a test fails, you know why it failed?

Vladimir:  One point here is to make some kind of login in your tests so that if a test fails, you can easily know what happened without even trying to run it on your machine. I think we just employed to have a login in our project. It didn't work out in every case but it did work in many of them. If it didn't work, we just ran it on our machines and tried to see what happened. Because the tasks are pretty reliable, the issues that our [TFS 00:19:32] or test can see if this is compound, they were producible on our machines as well pretty quickly.

Joe:         Awesome. Now, White, it is an open-source project. I think sometimes people sometimes don't always want to jump to an open-source project if it's not active. Do you know how active the community is for WPF? If I had an issue, do you know is there a community or a forum I can go to get good feedback on ways that I can possibly troubleshoot my issue?

Vladimir:  Yeah. There is actually a good community behind White. The White library itself is from TestStack which is a group of developers who focus on automated testing for [different 00:20:09] projects. They have other amazing projects, for example, BDD5, which helps convert unit tests into BDD tests. Another one is Seleno, which works on top of Selenium and gives some additional benefits. For example, it allows you to generate strongly-typed models. The community support for White is pretty good.

Joe:         That sounds awesome. I never heard of either of those other tools. One thing with White, I did a few things on the side with it just to learn it. I didn't find many resources on it. Do you know? That's why I'm so glad to see your video within the Pluralsight course library now because I think it actually fills a void that I think has been there for a while. Are there any best practices you can use with WPF to create more re-usable automation with it?

Vladimir:  I would say the best practices here are the same as with other automation activities. I know, Joe, you talk a lot about it in your podcast but I want to just reiterate it once again. First of all, don't use a recorder unless your test suite is really small and you're not planning to expand it. Secondly, that all should be a layer of abstraction between your tests and the White library. In the course, I name it the main specific framework. I heard you called it [automational 00:21:30] framework. Whatever the name is. The presence of such layer is vital to the maintainability of your tests.

Finally, strive to achieve the highest level of simplicity possible. In my Pluralsight course, you can see that the code of the domain-specific framework and the code of the tests never get [inaudible 00:21:49]. That's for the details. It is small and concise. Although a real-world application would be more complex than that, in my experience it is possible to keep the same level of simplicity, even with large projects.

Joe:         That's a great point. Also for re-usability, the less code you write, the better off you're going to be because maintaining code is a problem. I definitely agree with that approach. Keep it simple. Create layers. So you don't repeat yourself, just use the dry principle. I definitely agree with that. Vladimir, I know you cover this in your course. I highly recommend everyone can go to Pluralsight. I guess you can try it out for free for 14 days but when would you invest in automated UI tests?

When do you think it's a good idea for a team to actually use automation because it is a development effort. It takes time. You have to maintain it. When do you think it's a good investment to say, “Yes, this project probably could use automation,” and it's worth tjeor time and effort?

Vladimir:  Sometimes when a new project is started, software developers and even project mangers get too enthusiastic about automating UI tasks. They start to invest in them right away from the beginning of the project. I think we need to be pragmatic here. If a project is just started and it doesn't take long to run the regression test with manually, there is no need to put your efforts into the UI test automation. I think it's just fine to let the project grow to some point where automated tests really provide significant value. The reason here is that when a project begins, it's much better to focus on its functionality rather than thinking about UI automation or prompt.

It's also a good idea to make sure that your project is going to be big enough because it doesn't make sense to invest in test automation on a small project. Automating UI tests is a big investment. It doesn't pay off in small projects where it can confine itself to unit testing.

Joe:         Vladimir, are there any books or resources you would recommend to someone trying to learn more about test automation using White or WPF?

Vladimir:  I definitely recommend to subscribe to Pluralsight because it's an incredible source of knowledge of all kinds but that's an obvious answer, especially from someone who is authoring for Pluralsight.

Joe:         I love Pluralsight. I'm not an author yet but I definitely agree with you there.

Vladimir:  Another resource I recommend is actually a book, “Domain-Driven Design” by Eric Evans. It targets software developers primarily but I think most of its concepts are applicable to test automation as well. If you want to create a great abstraction layer for your tests, a great domain-specific framework or automation framework, you need to read this book. It covers all the concepts I think any automation framework should use.

Joe:         Awesome. It sounds like an awesome resource. I think I've heard a few guests recommend this book. It's been on my reading list for a while. I think I'm actually going to start diving in because everyone's been recommending it. Thank you for that.

Vladimir:  That's actually my favorite book I ever read in programming.

Joe:         Wow. I definitely want to check it out then. That's awesome. Vladimir, before we go, is there 1 piece of actionable advice you can give someone to try to improve their White WPF automation efforts? Let us know the best way to find or contact you.

Vladimir:  Read more about software development and best practices. Practice them as much as possible. It will pay off greatly over time. [Inaudible 00:25:17] technology gets outdated pretty quickly. At the same time, the principles of software development live much longer. You can contact me at my blog, EnterpriseCraftsmanship.com. I place all my contacts there, so it's a good starting point.