Welcome to Episode 71 of TestTalks. In this episode, we'll discuss the how Python can help test embedded systems with Brian Okken author of Python Testing with UnitTest, Nose & Pytest, and host of the new Python Test Podcast. You'll learn some tips and tricks on how to better test headless applications.
Is C++ the only program that can be used to test embedded systems?
With the release of Micro Python, which is supposedly super fast, that may be changing soon. The industry as a whole might have to reconsider the notion that C++ is a requirement for every fast interaction. Discover how the Internet of Things and raspberry pie might actually explode the use of Python for Embedded Systems.
Are you ready? Listen to Episode 71 of TestTalks with Brian's interview to find out more.
Listen to the Audio
In this episode, you'll discover:
- Why TDD can improve your development efforts
- How Python can help automate embedded programming tests and more
- What are PyTest and nose
- Tips to improve your Python testing efforts
[tweet_box design=”box_2″]The beauty of testing is that it gives you the freedom to refactor without fear~@brianokken [/tweet_box]
Join the Conversation About Python
My favorite part of doing these podcasts is participating in the conversations they provoke. Each week, I pull out one question that I like to get your thoughts on.
This week, it is this:
Question: What's your favorite use for Python? Share your answer in the comments below.
Here’s the Full Transcript of Python for Embedded Systems
Joe: Hey Brian. Welcome to TestTalks.
Brian: Thanks.
Joe: It's great to have you on the show, but before we get started, could you just tell us a little bit more about yourself?
Brian: I'm a software engineer. I'm mostly a C++ developer. I got a computer science degree in '96, a master's, and jumped into the hardware world in about, I don't know, around 2000. Started learning about test-driven development, and using Python for our testing, and went from there. I work in the test and measurement industry, making oscilloscopes and things like that.
Joe: Oh, cool. I guess what first strikes me about this is that you are a real programmer, C++. That's hardcore programming, I think, and you're more from, I guess, my generation, where most developers that I speak to, they don't like testing. Anytime they hear testing, they get completely turned off because they automatically think of QA testing.
How did you get into testing, or why do you think testing is important?
Brian: I've worked in a lot of different groups, and sometimes we've got a QA team, but I've worked on a lot of groups that had no QA team. I have seen that reluctance to test, but just things go way better if you've got dedicated testing in your team. Actually, it's surprising to me, because I think the lesson we learned from test-driven development is that testing is really, really important, and it makes everything go easier if you figure that out first. If there's no QA team, there's nobody doing that, so I think developers have to be involved. I'm not sure how I got involved in it too much. Mostly when I started … Actually, one of the difficulties when I started learning about test-driven development, there was this focus on, and there still is a focus on, the unit test model. That's really difficult in an embedded environment in which I'm working.
It was easier to morph the model into more of a functional … I thought of a unit as a unit of functionality instead of a unit of code, and that seemed to work fine. Also, I was testing the same API our users have, so I think a better focus for developers is to focus on the user-level API, and that brings in … I've also seen a model where developers are actually, especially in the hardware industry, and with our age level I guess, there's a lot of teams that they went through layoffs around 2000 and then again, I don't know, a handful of years later. Often the QA team was the people that got laid off. That's one of the reasons why people are reluctant to put testing on their resume, which is unfortunate now. I don't know if that's still the case, but anyway, it's just easier to develop when you've got solid tests in place.
Joe: Awesome, so can we just expand on that a little bit more? What kind of security does testing give you with your test when you're developing? Do these tests get in your way, or do you have an example of a time when your tests actually helped you find an issue before you released it to the public or something like that?
Brian: Yeah. Actually, almost every day. Especially, I'm a team leader now, and what I really need is hardcore C++ talent on my team. That's what we're doing. I often have to look outside of the electrical engineering world, so nobody on my team has an electrical engineering degree. These instruments are really complicated, and to make sure that we really understand the user interaction, we really aren't because we're often not the target market. Little corner cases, like you've got features that are in place that are really important to some customers, but on a day-to-day basis, we don't use them. Those things, having a test suite to make sure everything works, it's the only way to go.
Just the other day, I re-factored something to make the Python code look nicer, but it swapped around a couple of industries. We've got a whole bunch of different channels within a protocol that the numerations aren't lined up between the hardware numerations and the user interface numerations for what channels are which. They all look the same, and without tests in place, I mucked it up. The tests helped me to make sure I didn't deliver that to customers.
Joe: Awesome. I like the example you just gave, how you re-factored something just to make it look better. It's almost like the boy scout rule. If you see something, and you think it could be better, make the change, and everyone's better for it, I think.
Brian: Yeah, and that's a wonderful freedom that testing gives you, is the ability to refactor without fear.
Joe: That's a great point. Can you tell us how testing or developing an embedded system is different than testing for, say, a web application? Is there anything different that a developer needs to be aware of when they're testing and developing an embedded system?
Brian: I'm not actually that experienced with web development, so I'm not sure what the comparisons are. I've done a little bit of designing a website or something, but nothing professionally. One of the big things is, for instance, we can put unit tests in place in an embedded environment, but what happens often is, those will be put in, and they have to be compiled into a different executable, so it's a completely different environment, and they're often run, the unit tests are often run on a PC instead of in the embedded environment. I think it's more important to focus on the environment that the user ends up actually seeing, so I think the functional-level tests are … They're easier to work with.
There's a lot of difficulties with developing embedded that I guess you don't have to deal with in other places. It seems like actually that stuff would be hard in web development too. You're either working on the live system, which you don't want to test on and often there's only one live system, which I don't have that problem, I can stick test code on my instrument, and it's not going to affect customers. Actually, web development and web testing seem a little harder to me.
Joe: I guess when I think of embedded development versus web, just from a testing perspective functionally, you can do an end to end testing, people understand how to open up a browser and do a workflow and automate it, but for an embedded system, I would think that would be not as intuitive. I guess you're just directly interacting with the APIs, and anticipating what a typical workflow would be, connecting different APIs together?
Brian: Yeah, well, like most of our … At least all the instruments I've worked on, they all have a remote API, and that is by far the easiest thing to test against. Because of that, it wasn't always the case, but a long time ago the entire industry, I think, moved towards a model where the graphical interface, and the hard keys on the front of the instruments and things like that, they interact with something like a database or something. [inaudible 00:07:20] API interacts with the same database, so the entire system, if you test it through the remote API, you're going to hit 95% of the functionality that the graphic interface hits also. I've looked at some people, some people within the web world have a model that the graphic interface and the web interface need to be built on top of an API. That seems cool, that's basically what we do.
One of the interesting things about test instruments is you don't really know how the customer's going to use them, and you can't get live feedback or anything because of security concerns. Test instrument customers aren't going to want you to stick some cookies on their machine that sends back information on how they're using their instrument.
Joe: I see how that could be a challenge. I guess you just need to just test the best you know, and based on feedback from your customers, maybe over time, you get some sort of knowledge of knowing what type of scenarios you might want to definitely touch or test.
Brian: Yeah, and our teams are filled with a handful of really seasoned and experienced electrical engineers that work with the customers. We have application engineers that work closely with the developer on the customer side, and those guys sit close by. We can talk with them to figure out how the users are using things.
Joe: I just want to switch hears really quick. The main point, or the main topic I've really wanted to talk to you about, was Python, and the reason why I came across your Python test podcast, and it's all about testing with Python. How did you move from C++, or how did Python come into the picture in your career?
Brian: I was talking about the remote API for the instruments, and it's a text-based interface that's really well-defined. Actually, there's an industry standard called … I'm going to get this wrong. SCPI I think, SCPI, that we built our interfaces for instruments, and almost any hardware company does the same. These, you can use any programming language to control. I don't know, was it the early 2000s, I was on a team where they decided to use Python for writing tests against the instruments. There was this custom framework that they had, so I just learned just enough Python to be able to write these little Python scripts within and test within the framework. I don't know maybe 6 years ago, I found myself without a test framework pre-written for me.
Oddly enough, instead of going out and looking to see what else was there, I just wrote my own and wrote a test executive, and wrote it in Python. Yeah, I love Python a lot more than C++ now. You can get so much done way faster. In our situation, I think it helps. The hard part is I need to convince C++ developers that they need to learn Python, but it's taking less convincing nowadays. One of the great things is that you have to think about the problem from the user's perspective in a different language than the language you're implementing, so that is a mind shift to think like a customer, versus think like a software engineer. The switch of using 2 different languages, I think, actually helps people to have that switch.
Joe: Wow, so that's a different perspective that I haven't heard before, so that's a great point. I think most people think that doing that context switching, you would be less productive, but it sounds like you're saying that switching to a different language gives you actually a better perspective maybe, that you may not have thought of if you were just using the same exact language as the software that you're developing.
Brian: Yeah, and also, there are so many cool things. We're dealing with often huge arrays of numbers that come out of these instruments, and being able to flip them around and manipulate them really quickly with the Python tools. It's something that it's a lot more painful in something as low-level as C++.
Joe: What you mentioned earlier, I would think would be an issue also, trying to convince, say a C++ developer to even consider using a scripting language like Python. Do you consider Python a real programming language that enables you to do real development tasks? When would you use C++ over Python, or what are your thoughts about that?
Brian: It definitely makes sense for … The C++ is just a requirement for hardware level interactions at the speeds that we're trying to do. We're trying to squeeze as much performance as we can out of our instruments. It wouldn't make sense to put … Well, I don't know. It might make sense to put an embedded language in there, or the Python or something running the instrument, but it's just generally not done still. C++ I by far the most widely used language when you're talking directly with hardware. There's probably a ton of C++ code in every car probably with all the chips that are running in there, but anything outside of talking directly to hardware at a fast speed, I think Python makes sense.
I have been paying attention to a lot of the … Micro Python has come out, and it's supposedly super fast, so we may be changing soon. Not our company, but I think the industry as a whole may be having to reconsider that notion that C++ is a requirement for every fast interaction.
Joe: I wonder what's mking that push. Is it more like the internet of things? As more and more things get embedded into chips that actually embedded technology, or embedded developing and software testing is probably going to grow over time?
Brian: Yeah, definitely. I think the internet of things, the raspberry pie and the other small boards to let people tinker with stuff has actually exploded the use of Python for hardware. That's pretty cool.
Joe: Awesome. What's your view on test-driven development? I know you have a blog post on this, I just wanted to know what your views were. Is TDD dead, or do you use TDD all the time?
Brian: Yeah actually around the time where I was really trying to come to grips with when I started writing this blog, I wanted to write more about processes like test driven development, so I went out and did a bunch of research and a lot of people writing about TDD were writing about something that I wasn't familiar with. This isolation model, isolated unit test using mocks and stuff, and that's jut really foreign to me. I think of tests from the customer's perspective first. I definitely think that tests around important sub systems, we've got some really hardcore, some real algorithmic stuff. If it's really some core algorithm, of course you're going to put some tests right around that, but at a functional level it doesn't make sense to me.
I thought I had just learned TDD wrong the first time, but then with that TDD is dead thing that came out, they got all these people that were near the beginning of tester and development, and they were thinking the same thing. Don't go crazy with these isolated unit tests. Hearing some people say they've never used mock and probably never will, I think that's great. It enforced the fact that I'm not crazy with thinking of the entire scale from functional tests down to unit tests as an as needed basis. You don't have to go crazy and test everything.
Joe: Brian, are there any cool or must-know Python libraries that you use all the time that you think everyone should know about?
Brian: Well, NumPy opened up quite a lot of our world for us. We're actually slow to pick up new things, because the core language itself gives us quite a bit. That's probably a no-brainer for most Python developers, but it's good to pick up some of these scientific computing things, especially within the electrical engineering testing.
Joe: Very cool. Brian, I know you also wrote a book on Python testing with unit tests, nose, and pi tests. Can you just tell us a little bit more what exactly that book covers, or where people can get a hold of that?
Brian: Yeah, so at pythontesting.net/book, I've got it there. The impetus of the entire website was that I had written this test executive from scratch, and then went out and looked, and found unit tests, and nose, and pi tests. I wanted to compare all 3 and really figure out how each of them worked, and if I should switch to one of them. I couldn't find any reasonable tutorials that were fair to each other. There were some nose tutorials that slammed unit tests, but some of the stuff didn't apply anymore. I really wanted to create a resource that could treat all 3 of them fairly, and go through a similar model, so I came up with this silly subset of markdown, just a few pieces of markdown, and started writing a script to convert markdown to HTML, and then wrote some tests around it and tried to do functional level testing in all 3 of those.
I also started with DOC tests too, but I got frustrated with DOC tests really, really fast. That's really the core of that book, is just all of those introduction … I wanted an introduction to all 3 of them that had the same flavor, so that they were pretty easy to compare the three, and then extended them into using fixtures a lot, because fixtures are really important in my work. I think fixtures are something that's underused in a lot of Python tests.
Joe: Awesome, I think the book's a great resource. Also I think an episode, your second episode you covered this in your Python test podcast, pi test versus unit test versus nose, that people should check out. Is there one thing you see over and over again in your experience that you think most people using Python are doing wrong?
Brian: In my world what I see a lot of times is people using it as C++. That's not really in the Python world. One of the problems with using multiple languages is, and a lot of them are based on the C Syntax, it's easy to just use a new language and do everything you did before, but in the different language. I think it's really important when you're picking up a new language to actually pay attention to the differences, like the 4 loop is completely different, so why is it different, and try to embrace the differences instead of working around them. A couple years ago embracing the idiomatic Python, or Pythonic way, there's tons of posts and a couple books, I think, on that, and getting a little bit into that to make sure that you're doing things as concisely as you need to be.
The other thing that I see a lot is people not looking for … There's tons of standard libraries and things on [inaudible 00:18:49] that people don't know about. Especially with a bunch of engineers, and this may be true everywhere for Python developers, it's often a lot easier just to write your own than to look to see what the best in the industry is already.
Joe: That's a good point, and I see that all the time with Java developers. I think that's a common theme across all languages with developers. Brian, are there any resources or books that you would recommend to help someone in their Python testing or Python developing efforts, to get them up to speed?
Brian: Actually, I keep getting asked that question within my company, and that's one of the reasons why I'm writing about it, because I don't have any good resources. Harry [Persible's 00:19:31] book I think seems pretty decent if you're a web developer since it talks about Jango testing. I'm probably not a good person to critique that, since I'm not a Jango developer. One of the things that it's probably not really Python, it's probably all developers, is this notion of, “Management wants me to do something,” Or, “Management doesn't understand,” But there's a little book out there called Team Geek. It's a really quick read, and I recommend everybody read it, because it helps managers think like developers, and it helps developers think like managers. I think everything runs more smoothly if the engineers think a little bit more like a bean counter.
Joe: Cool. Sounds like an awesome resource. I love books, so I'm definitely going to add that to my reading list. Brian, here's a random question. Just popped into my head. Is there a philosophy or a point of view you have towards Python or Python testing? I tend to find that people that use Python have a certain way of looking at things.
Brian: I'm from Oregon, and we … I don't know if it's just us, but we think about things a little differently. There was a t-shirt that I saw in college that said, “Conventional wisdom is seldom both.” I actually think like that a lot now, and so there's … When you get to the point where somebody says, “Oh yeah, we always do that because it's just the right way,” I think it's important for everybody to at least shine some light on it and make sure they understand why it's the right thing to do before they just go off and follow what the conventional wisdom is.
Joe: Okay Brian, before we go, is there one piece of actual advice you can give someone to improve their Python testing efforts? Let us know the best way to find or contact you.
Brian: If you're not already using setup and tear-down fixtures, and go ahead and use those. Don't be afraid of classes within testing. Then the last thing I think is, it takes a while to get your head around PyTest fixtures, but it's well worth the effort because they're extremely powerful. To get a hold of me, you can find me at pythontesting.net. I'm also @BrianOkken on Twitter.
Want to Test Talk?
If you have more questions about Python implementations and testing, comments, thoughts, or concerns, you can do so by visiting our website. I'd love to hear from you.
So, how to get promoted on the show and increase your kama?
Subscribe to the show on iTunes and give us a rating and review. Make sure you put your real name and website in the text of the review itself. We will definitely mention you on this show.
We are also on Stitcher.com, so if you prefer Stitcher, please subscribe there.
