Welcome to Episode 91 TestTalks. In this episode, we’ll discuss test automation with the creator of the Robot Framework, Pekka Klärck. Discover the reason Pekka created the Robot Framework as well as how it can help you with your testing automation efforts.
The Robot Framework uses a keyword-driven approach to make tests readable and easy to create. It also has many test libraries and other tools that you can use. The Selenium WebDriver library is probably the most used external test library, but Robot Framework also can test FTP, MongoDB Android and more! In addition to all this open-source awesomeness, it has a lot of API's to help make it as extensible as possible. The Robot Framework is Python based, but you can also use Jython(Java) or IronPython(.NET).
So if you haven’t yet tried the Robot Framework, it may be just the testing tool your QA teams have been looking for. Discover how to get started with Robot Framework in this week’s TestTalks episode.
Listen to the Audio
In this episode, you'll discover:
- Why Pekka created the Robot Framework.
- What languages and test management tools Robot Framework supports
- How the Robot Framework is different than other software testing tools
- The top 7 things you need to know about the Robot Framework
- What Pekka's main test automation tip is and how the Robot Framework can help
[tweet_box design=”box_2″]The @robotframework uses a keyword-driven approach to make #tests readable and easy to create[/tweet_box]
Join the Conversation
My favorite part of doing these podcasts is participating in the conversations they provoke. Each week, I pull out one question that I like to get your thoughts on.
This week, it is this:
Question: What are your thought around keyword driven test automation frameworks? Share your answer in the comments below.
Want to Test Talk?
If you have a question, comment, thought or concern, you can do so by clicking here. I'd love to hear from you.
How to Get Promoted on the Show and Increase your Kama
Subscribe to the show in iTunes and give us a rating and review. Make sure you put your real name and website in the text of the review itself. We will definitely mention you on this show.
We are also on Stitcher.com so if you prefer Stitcher, please subscribe there.
Read the Full Transcript
Joe: Hi Pekka welcome to Test Talks.
Pekka: Hi, great to be here.
Joe: That's awesome to have you on the show. Today I would like to talk about your awesome framework, the RobotFramework, but before we do, could you just tell us a little bit more about yourself.
Pekka: Oh sure, my name is Pekka Klärck, and I am independent consultant from Finland, specializing in testing, test automation, and Python programming, and of course RobotFramework. I started my career in testing, 2000 right after millennium and I have been working in testing and test automation since that. I was working with various kind of test automation tools, writing different kind of frameworks using Pearl and other languages until 2005 when I started to work on my masters thesis, where I actually came up with the idea of RobotFramework.
Joe: Awesome. So what made you start thinking about this pretty masters thesis. What made you want to create the RobotFramework?
Pekka: Initially when I started it was not that, okay I am going to create this kind of framework. Of course my main motivation was to graduate. I had been in the university quite a long time I had been working long time and studying math sand I was like now I need to get out there I need to masters thesis I had been developing several frameworks, for data driven frameworks, and some [inaudible 00:01:21] frameworks, and I had noticed that there's quite a bit in common with them. I decided, I will study that, and see what the literature talking about them and see if there could be certain common components like process, or log creators so whatever that could be used when creating such frameworks in the future. Actually the biggest finding in the thesis was that you can have a generic framework so you do not need to have like separate process, or separate drivers, or things like that. You can just have general framework and you can be using that in various environments, and various contexts. That was going to be pretty cool
Joe: Awesome, so it sounds like it has been around for 11 years now? So it's a pretty mature framework?
Pekka: The first prototypes were done I think 2005 if I recall correctly for the masters thesis, and then I got very lucky. I had a colleague from an old project called me, and he had signed Nokia networks, and they needed test automation there, in an environment where it was not possible to use any kind of pre-made, so on. We had had good success in an earlier project and he asked me that, would I have any kind of ideas what should be done and well I have just been doing this kind of prototypes in these are quite promising. We could take a look at this, and that is how it started. We started to create RobotFramework from scratch, not using the prototypes, but using those ideas. It was used several years at Nokia networks, internally. My idea and also everybody else's idea was we would like to open source as soon as possible but, as you can imagine in that kind of corporation, it involves lawyers, so as soon as possible was like three and a half years, but it was finally opened I think in 2008. It's been available publicly since that.
Joe: So there is a lot of frameworks out there. How does RobotFramework compare to other frameworks? What makes you unique, that you think that someone that starting automation would want to look at this particular framework?
Pekka: Yes, you are right that there are many frameworks that are very similar. For example the kind of frameworks that people are most commonly comparing RobotFramework are obviously Fitness, and then Cucumber. When RobotFramework started, Fitness existed, and actually Fitness was tested but coming up, there was a FIT, was around that time and Cucumber didn't back then. Comparing to Fitness or FIT, RobotFramework is more close to FIT in kind of how you use it than Fitness which incorporates with Wiki. The main difference is that you are not tied to having your libraries that we call them test libraries, and in Fitness language I think they call them fixtures. In our frameworks you basically have one way to create libraries. You have libraries that provide keywords. In Fitness you have fixtures that are implemented differently depending on, is your test case an kind of a [wearcloak 00:04:32]. Werecloak kind of best case, or is it data driven style test case, and so on. So that was the main difference there.
With cucumber, I would say that the main difference is the original, or where they are coming from. So Cucumber is, well of course I am not Cucumber developer, so I hope somebody corrects me if I am talking something totally stupid but, my understanding is that it comes from, more from a developer business people communication. It is a tool for helping developers make the right product, and of course it is also automation tool, but it wasn't, that is how I understood, it wasn't initially designed that much to be a test tool for test engineers to use. Of course it can be also used in that domain. Whereas our framework initially was designed to be a tool for testers to create automatic tests, so that you could also have testers who do not know about programming, creating new test cases and so on. Of course automation always requires programming when interacting with the system and the test with those reusable libraries, you can create test cases also without programming knowledge. Well base, RobotFramework has been extending also to other areas, partly, inspired by what Cucumber is doing. Nowadays you can also have that high level test cases, given when then kind of Gerkin's variation. Also with RobotFramework, if you need that. In robot framework you are definitely aren't forced into that kind of system. You can also use other ways to create test cases.
Joe: I think it is a great point that Cucumber really was created for communication between developers, product owners, and testers, and RobotFramework from the beginning was created as an automation tool, but also given you the option to write your test in a clear English like syntax that anyone can read, so it is somewhat the best of both where you have the clarity of what the test is doing in an English like syntax almost, and it was created for automation. I do not know if you agree with-
Pekka: Yes
Joe: -that or not, but I think that is how I took it as.
Pekka: That was the idea. Of course the idea has been all the time to be able to create as clear and as easy to understand test cases, that of course is well if nobody can understand your test the test is pretty much useless, but also that typically enhances maintenance which is very very important in test automation.
Joe: Awesome. So for the beginner that has never heard of robot framework a lot of people may think of automation they automatically think of Selenium. Is the robot framework just for Selenium or does it also handle other types of technologies like API Testing and things like that?
Pekka: Definitely not only for Selenium. Another thing actually, if you compare RobotFramework to those fitness and Cucumber, one big difference is that because of the test library API design, because the idea is that you can create generic test libraries. They are a lot of libraries available, so the wider ecosystem around RobotFramework has lots of test libraries and also a lot of other tools that you can use, and Selenium library actually Selenium 2 library nowadays, is probably the most used external test library, but RobotFramework is not only for that. You can use it for anything besides, I know project where it has been used to test kind of a, diesel engines, and they do not have web services there. It has also been used to test kind of, other kinds of embedded systems, networks, different kind of network elemental builds, we have Nokia networks, and so on. You can use it also with the Selenium library to do web testing and you can use it with Swing library to do Swing user interface testing, and you can use it with database libraries to test databases, and very importantly, you can use it with those various libraries in the same tests, so you can have a test case that is interacting with the web page, interacting with Swing user interface, and also interacting with the database.
Joe: Now that is another great feature of the RobotFramework. If someone is starting with Selenium from scratch and they're building their own framework, for example if you work for a large enterprise, a lot of times you need to test things that are not just web applications, and what is great about the RobotFramework is like you said, there's these other existing libraries already that you just need to pull into the framework, and you are able to access some existing functionalities, some existing keywords, that will allow you to interact with things other than a web page. I think that is a big feature of the framework.
Pekka: Yeah, that's definitely been the idea from the very beginning, but of course the idea has also been it should be easy to create your own library so when you need something custom, be it your custom programming API, or custom some other API, or you have custom need with the browser for example, so it is easy to create your own libraries, and Robot's libraries API is very straightforward and simple, so that is very trivial. That requires, of course, programming, although there idea is that you can create test cases without real programming knowledge when you are creating libraries that is of course mandatory if there is no way around that.
Joe: Absolutely. That is actually one of the question I got asked by one of my listeners. Rob asked me. The question is, “How much programming knowledge should someone have in order to get started with the RobotFramework?” It sounds like you could have maybe an automation engineer that knows program that sets up the main libraries in the main framework itself, but they create it in such a way that they use keywords that can then be used by a person that does not really know programming to create tests. Is that how it would work?
Pekka: That is exactly how it would work. To get started you probably need somebody who understands a bit more about test automation and programming, and in any kind of larger team, it is likely that you are going to have problems if nobody there knows how to extend the framework with custom libraries, or to debug problems with existing pre-made libraries. Normal users can easily use it without programming knowledge if they are introduced to how the framework works, what keywords they have available, and so on. One kind of unique feature in RobotFramework is, I have ability to create higher level keywords which we also call user keywords, using the same test syntax that is created for … Although it is used for creating test cases. Your library can provide very low level keywords like Selenium library has keywords, open browser and click link, and page with content and stuff like that. Based on those, you can create your own high level keywords, like open browser to login page, or input username, or stuff like that. Something that forms your own domain specific language there, and using those keywords it is very easy then to create test cases.
Joe: Awesome. So I guess another question I have is when people hear of Selenium once again they think of all the different language bindings you can create your framework in. For the robot framework, what language bindings does it support? Is it just Python, or how does someone go about creating a test within the RobotFramework? Do they have to choose a language, an IDE, or how does that work?
Pekka: It of course depends on what kind of interface you are using when testing your application or interacting with the application. If it is for example web app, then of course the language RobotFramework uses, or stuff like that isn't very relevant, but RobotFramework itself is implemented using Python. Python is the main language to use when extending the framework also. So you can majorly create test cases using Python. RobotFramework also runs on Jython which is a Java implementation of Python so it runs on JVM, and on Iron Python which is a .NET implementation that runs on obviously on .NET. Using those versions of Python you can then majorly interact with Java or .NET APIs. It is even possible to create when you are using Jython, it is possible to create test libraries using Java so that you do not need to write them in Python at all. Unfortunately that is not possible at least at the moment using C sharp, but it would be pretty easy to create them. Simple Iron Python or Python wrapper on top of C is our library. Hopefully at some point, we get the full request or I have time to implement this C sharp library natively in the framework, but at the moment that is not yet possible. There is also so called remold library API that basically allows you running test libraries as separate processes that may be running even on a different machines and then you communicate with them using so called remote library API that allows running test libraries as separate process, that might be running even on a different machines, and then you communicate with them using a so called remote library interface that is implemented on top of XML RPC.
Joe: Are there any tools that you use all the time with the robot framework that helps you create better tests? Like what's your favorite IDE for creating testing?
Pekka: There are a lot of different options when you are creating tests cases, lot of editor options. Probably the most well known is RIDE which is a stand alone the Robot Framework editor, only designed for editing RobotFramework test data. I do not personally use that too much because most of my RobotFramework development involves Python coding, and RIDE is only for editing RobotFramework data but you cannot edit libraries with that. For me, I happen to like PyCharm more, IntelliJ based editors for Python programming. That is just my preference. There also Eclipse which are good too. With PyCharm I don't use the provided RobotFramework logins that give me some help like index highlighting, and there's some kind of navigation from keywords to implementation. Even some kind of completion-ment, like key word completion when pressing control space. RIDE does stuff like that much better. It has also support for renaming keywords and all stuff like that, but for me, because I need Python programming there at the same time. It is easier typically, to use PyCharm or something like that. There are plugins for PyCharm, there are plugins for Eclipse, IntelliJ, and also for many popular programmers text editors like NotePad++ and TextMate and Sublime Text, and so on.
Joe: Very Cool. I guess if someone is starting off fresh brand new once again, like I said, I never really used the framework, I tried it a few times. We use a Java framework that we have already created for our company, but this is a tool that I think could help a lot other companies and a lot other groups within my company, so I am really excited about it. When I think of it, like I said, I don't have that much exposure to it. I just think of it as a keyword driven framework that you can data drive easily with Excel sheets. I don't know is that's a correct interpretation of what the framework is. Is that how you would explain it so that someone doesn't know the RobotFramework?
Pekka: How I would explain it would be highly dependent of course what that person knows about test automation, and testing in general. Does that person for example know Cucumber and Fitness? At least that is true, well it's, definitely it's a keyword driven framework. The basic idea that you construct your test cases from keywords, and keywords basically map to some action that you are going to do. Action can be doing some kind of set-up activity, it could be doing something for the system or it could be validating the results, anything like that. The simplest case is that you create a test case that contains the workflow using those keywords. Do something for the system first, then you validate the results. Then, if you would like to move to this data driven style where you are not so much interested in the workflow, what happens, what more interested in data when you pass some information in the system will get right results out, then you would switch to data driven mode.
Which also works very nicely with RobotFramework although it is implemented on top of keyword driven. It's implemented on top of keyword driven approach. That was actually one of the findings that I realized when doing my masters thesis back then was that, data driven testing, I knew that it has some really good use cases where you have like tables where you have columns containing different input and output values, but that you can quite easily actually implement that on top of a generic keyword framework. You don't necessarily need to have a separate data driven framework and separate keyword framework. It is enough to have a generic keyword driven framework and then implement our data driven test also, on top of that. I think that if I would really would be showing RobotFramework to somebody, I would probably go with examples. At least show some concrete examples, and preferably if I would be from my laptop, I would then of course run some example test cases and show reports and log files, and so on.
Joe: Something which is very flexible. You could use it, however, it does not sound like it is going to force you to create your test in a certain way that maybe against the way you may be used to within your company already. Is that a true statement?
Pekka: That is very correct, and that is both pro and con. It is very flexible, and it is actually so flexible that sometimes it is hard for people to get started with, because there is no single way that “Okay, you start with this.” Of course in any kind of a bigger project, it's very important that it doesn't force you to do that many things in certain way. Of course there are certain rules, but it tries to be as flexible as possible, and as you just said, the idea is that you should be able to adapt that to your current way working, rather than adapt your current way of working to what the tool wants to do. That is very important.
Joe: Great. We need you to know that RobotFramework has a lot of libraries out there. It's not just for Selenium. I just want to re-iterate that. It works with API's, I assume it works with things like Appium. My next question is, does it work with other tools like, say, TeamCity or Jenkins, or TestRails, or Quality Center? How does it integrate with other tools like test management tools and CI systems?
Pekka: [inaudible 00:19:39] creation story's pretty good. First of all, even without any kind of special plugins, you could just run it, because you can execute it from the command line. All CI systems support that. And then, from the command line, you even get the written code to the system basically, tell you the overall very high level status. Then you get logs and reports, and if you can show them from your CI system, then that gets you quite far. With Jenkins, we have a separate plugin, and that is pretty good. It gives you statistics, shows you trend charts, how many test cases you had last time, how many of them were passing, and so on. That is pretty handy. It's available as a standout Jenkins plugin, so you just go to plugin manager and install it, and configure, and you're good to go. That works pretty well.
Then, for test management system in the creation, you basically would need to have some kind of plugin that allows you to push information there, or something like that. I know that there has been something for TeamCity. I know that at Nokia they have had, they've done something with QC, Quality Center, but I don't think any of that is open source. It's definitely possible. It's not that widely used as the CI integration, because everybody's using CI if you want to get your tests run automatically. Especially in more agile setups, people might beat us. Might not use Quality Center or other test management systems that much, instead it might be enough that they have those test cases in easy to read form, and available on a file system. They maybe don't need much more than that. I know there are places where management systems are used, but I cannot really comment on what's available at the moment.
Joe: Sure. Like I said, it is extensible, the RobotFramework. It's not like if it doesn't support it, you can't do anything about it. I know the newest versions of Quality Center ALM has a rest API, so, you can do a rest API and that should be fairly, something that can be done with the RobotFramework if you wanted to extend it.
Pekka: Exactly. One important thing is that when you run test cases, you get results in XML format. If you have a system, a management system you can just parse the XML format, either using any XML library you want to, or perhaps using Robot's internal result parsing modules. Then based on, you can push information through to the system however you want to. However that system supports it. All information about test execution is very easily available. In addition to being available, after execution in that XML file, there are also the so called listener API, listener interface that allows you to monitor test execution while tests are running. That can also be used to push information to external systems if needed, if you needed information already during execution, not only after. There's a lot of interfaces like that.
This actually reminds me that, one thing to understand about the Robot Framework is although it is itself a framework, very often, when you use it in organizations, you actually implement your own framework on top of that. Your own framework that has has RobotFramework as a core, has several libraries probably there, maybe some custom libraries. Has your way to start test execution, has your guidelines for how to tag test cases for example, and how results are published, and so on. RobotFramework has a lot of API's to make that easy. The idea is it should be as extensible as possible. We've been adding more and more API's over time. For example, I would really like it to be easy for someone to create some kind of model based testing tool that could have test data in totally different format, but then it could still use the Robot Framework as the runner to execute test cases, and then would also produce normal logs and reports that RobotFramework produces. Stuff like that.
Joe: Are there any books, or resources, or tutorials you think would help someone trying to get started with RobotFramework that you would recommend?
Pekka: In general, I recommend going to RobotFramework.org. If you're totally new to framework, then just reading the initial introduction there, and looking at the examples that are provided there would be a good idea. Then there are several demo projects that you can study. You can download those test cases that contain the system on the test also, and you can download them, you can run them on your know machine and see how they work. That's probably the best way to kind of get a better understanding how it works. If you want to really start using it, other good resources, well, there are lot of resources listed on those RobotFramework org pages. There's a very active users mailing list. There's IRC channel, and Slack channels, and very importantly there's a very thorough RobotFramework user guide that you can use as the reference manual. You can also read it, if you really want to know the framework, you can read it, but it's over 100 pages long if you would print it out, so it's not small. It's not a small book. It's really good as a reference manual if there's some feature you, want to know really how it works.
Unfortunately there isn't any actual book introducing RobotFramework and getting started with that. I know there's some book, but I don't know, is that how good that really is, but that's been one of the things for me, that I'd like to do at some point. Is to write the book, but it's kind of problematic because I don't like writing that much, and I'm also not natively English speaking, so writing good English especially is quite slow for me, but maybe some day.
Joe: Awesome. That would be great, especially coming from the person who created it. I was going to ask, is your thesis available online for someone to read?
Pekka: Yeah, it's definitely available. It's not so long ago that I browsed through it, and most of it is still somehow relevant. Of course anything related to [inaudible 00:26:12] prototype implement testing there is totally outdated. Our current framework doesn't look anything like that. The concepts and ideas are more or less the same. It can be found through my company pages, eliga.fi. E-L-I-G-A dot F-I.
Joe: Awesome, so the next question I always get asked for some reason, is, because it's an open source tool, is there a place or a company, or an organization someone could go to if they needed actually to get consultant services to help them with their implementation, or issues that they may be encountering that they need more hands on help from someone outside the organization?
Pekka: First of all, those resources that I already listed, like the, our RobotFramework users mailing list, and IRC and Slack channels. Of course, there are a lot of people they're willing answer questions, and so on, on their free time, and they're very active. In general, there's a great community around the framework, but of course that's not enough if you need a professional support or training and so on. I know there are companies, especially in Finland, there are a lot of companies offering RobotFramework related services. I have a company that does that. I'm working as a independent consultant, and I';ve been organizing a lot of RobotFramework related trainings, and coaching, and reviewing test cases, and so on. I used to be working to consultant for Nokie networks, but my contract with them has ended recently, so nowadays I'm actually more interested in all kind of new kind of opportunities. You can contact me, but there's also others. In Finland what we're actually doing at the moment is that we are founding, we have actually already founded the Robot Framework association that is going to support RobotFramework development. It's going to collect membership fees from companies who are using RobotFramework and companies who are selling services related to RobotFramework, and of course all those companies in the latter category, serving services around that are companies you can also contact.
There was actually one thing that I realized here that I forgot to mention earlier. We've been discussing about many of the most important features in the Robot Framework, like ability to generate test libraries, and the fact that you actually have a lots of generated test libraries available. The possibility to create high level keywords, these user keywords to create your own domain specific logins. I briefly, at some point, I mentioned it's possible to tag test cases that is create, because you can statistics, for free. Based on that, you can include or exclude test cases based on the actions, and so on. That's pretty neat. Lot of that is also something many other frame tools have, but I think one really really nice feature in RobotFramework is that you get logs and reports that are both very management friendly, but also developer friendly. There's a report that as high level statistics about what happened and a lit of cases there, and it basically has red background if there's a failure, and green background if everything is okay, so you can use it basically kind of as a radiator. Then there's a log file that has all the details about execution.
That is pretty good. It's quite a lot better than having just askey, log, with tracebacks and stuff like that. Very important when you have a big amount of test cases, and more complex test cases, and so on.
Joe: It's actually a great point. Sometimes it's easier to create tests than actually debug them, especially when you have a large test suite and you run them in CI, and they start failing. Are there any built in reports or tools that makes it easier to debug or find issues when your test fails within the RobotFramework? That's what it sounds like. You make it very user friendly to look at reports and read logs.
Pekka: Yes. Logs work most of the time. That also depends how the library behaves. When you are using the library, it also depends how well that library is handling logging, and what kind of error messages you get from there. If the library is bad in that sense of course, then you don't get good logs. If the library is providing useful log messages, and error messages are clear and so on, then things typically are pretty easy. Of course, again, the lot of complicated situation also, where you need to debug something. The listener API for example that I mentioned can possibly used there. There are some debug tools available also in the ecosystem. Luckily they are not nearly that often. Typically, the ill behaving library can report everything so that you can easily see what is happening. The Robot Framework automatically makes it easy for you to see what keyword was failing, so that's not the problem, but then why it failed is something that the keyword itself should somehow communicate.
Joe: Before we go, is there one piece of actual advice you can give someone, to improve their RobotFramework automation efforts? Let us know the best way to find or contact you.
Pekka: My general advice when creating automatic testing, this doesn't only apply to RobotFramework but to any other automation tool, is to try to make your test cases as easy to understand as possible. Of course this somehow depends that if you are using programming language, you're using for example Java to create [inaudible 00:32:13] test, then it's not going to be as easy to read for business people, as a test case created using the Robot Framework or Cucumber is. Still, try to make it as easy to read and understand for people reading, for their kind of audience. That's kind of my main tip. Not really RobotFramework related, but of course very important with that too. The best way to contact me is to go to my company webpage, which I just mentioned. It's somewhat outdated at the moment, and I'm planning to re-implement them, add more information about the services that I do, but you can go there and contact me. The company pages are at eliga.fi, eliga being E-L-I-G-A, and agile backwards, and then F-I.
Comments are closed.