54: Al Sweigart: Automate the Boring Stuff with Python

31 May 2015, 10:48 AM

By Test Guild

Automate the Boring Stuff with Python

In this episode, Al Sweigart shares how easy it is for someone with little-to-no programming experience can get started automating some of the most boring tasks that fill our days.

What great about this is you’ll find automation can be used for much more than your typical end-to-end functional testing scenarios.

So if you’ve struggled in the past to learn a programming language for test automation, today you’re going to discover it’s much easier to get started than you probably think.

About Al Sweigart

AlSweigartHeadShot

Al Sweigart is a software developer and tech book author living in San Francisco. Python is his favorite programming language, and he is the developer of several open source modules for it. His other books are freely available under a Creative Commons license on his website http://www.inventwithpython.com/. His cat weighs 14 pounds.

Quotes & Insights from this Episode on Automating Boring Stuff using Python

Al's book will show you how to learn programming as a practical way to automate a lot of things that they would normally have to do with a bunch of clicking and typing.
Python has always been my go-to language for complete beginners. It has a really gentle learning curve.
With Python you can remember how to do all of these things in your head, and also it doesn't force you to learn object-oriented programming
Python would be a pretty good scripting language for people to learn in order to do basically whatever they need to do, and have the full power of Python and all of its modules behind it.
Pep8 is a Python document that goes into just the style guide, and that gives a lot of really useful hints.
Diversity is also a very important aspect. Not just for having the appearance of political correctness, but having diverse teams gives a lot of different viewpoints and input, into creating software
Much, much more!

Resources

Connect with Al

May I Ask You For a Favor?

Thanks again for listening to the show. If it has helped you in any way, shape or form, please share it using the social media buttons you see on the page.

Additionally, reviews for the podcast on iTunes are extremely helpful and greatly appreciated! They do matter in the rankings of the show and I read each and every one of them.

SauceLabsSponser

Special offer for TestTalks listeners, get 20 hours of automated testing for free when you sign-up with promo code testtalks14 (more info).

More about How To Automate the Boring Stuff with Python

Joe: Hey, Al Sweigart, welcome to the TestGuild Automation podcast.

Al: Hey, thanks for having me.

More About Al Sweigart

Joe: So before we get started, you tell us a little bit more about yourself.

Al: Well, I've been a software developer for about the past eight or nine years. I've been living in the San Francisco Bay Area for most of that time. About five years ago, I got started writing programming tutorials. A friend of mine was in au pair for this brilliant 10-year-old who asked her to ask me, you know, if there were any programming tutorials that he could find. And so she asked me to find someone I really couldn't find any that were good. I mean, the Internet has a lot of information that's out there, but it's all in different pieces. And so I just wanted to start from scratch and write something straightforward. Of course, that straightforward tutorial ballooned into a book-length, and eventually, I just self-published it, and people seem to like it. So I book-length wrote a second book and then the third book. And then, the fourth print book is Automate the Boring Stuff with Python. I left my software developer job last about a year ago and started writing the book full time. And it's just come out last few weeks, and I'm pleased with it.

Why Write Automate the Boring Stuff with Python?

Joe: Awesome. So actually, that's why I'd like to start off covering is talking about your book Automate the Boring Stuff with Python 2nd Edition from the Publisher No Starch Press. So what was the main reason for writing this book? I know you've written, I think, three or four other ones before this one, so why another book on Python Programming Language?

Al: Well, yeah. I've written three books before, like Scratch Programming Playground, Cracking Codes with Python, and Coding with Minecraft but they were all aimed at either kids or teenagers, or at least that's what I would tell people so that adults would think, oh, well, programmings for kids. So maybe I can do it, and they wouldn't be scared off. You know, those books were concentrated on games with Python and fun little crypto programs. I wanted something for adults with office jobs or adults who had jobs where they just use computers every day, which is a pretty substantial portion of the workforce. They could learn programming as a practical way to automate a lot of things that they would typically have to do with a bunch of clicking and typing. I would hear stories from my non-developer friends. They said, yeah, well, today at the office, I spent four hours opening PDF files, finding this one line that had some statistic in it, and then copying and pasting that into a spreadsheet and then moving to the next of hundreds of PDFs. And I would think, like, wow, you could probably spend about half an hour writing a script to do that instead of spending hours just doing it by hand. There are all these little stories like this, and there isn't any software written really to do these because they're very particular to an individual's workflow. But if you know to program and how to cobble together a custom script, then you could save yourself a lot of time. So I thought, well, there's got to be something out there already like this. And there isn't. There's a lot of books focused on turning people into six-figure making rockstar developers or for people who are already developers and programmers in different languages if you want to learn Python. It was nothing for complete beginners who didn't want to become software engineers or computer scientists and teach them just enough programming to be helpful.

Joe: Awesome. What I really love about this book, besides a cover, anytime I see the word automate and a robot on a cover, I'm usually reading that book. This book is for total beginners. But what I love is that it teaches practical things. So it's not like a hello world. You, how things like how to interact with Excel, read PDF, Web scraping, etc. So why would you recommend someone learn to do these types of things using Python rather than say a language like Java or C#?

Al: Well, Python has always been my go-to language for complete beginners. It has a gentle learning curve. It works on Windows, Linux, and Mac. And also, it's still a serious programming language that professional developers use. I think there's an XKCD comic that really extolls all the Python virtues and says, hello world is one line long and like Java where you sort of have to remember what Public static void string. Main bracket, right. Yeah. It's been a while since I've done Java programming, but yeah, with Python you can, you can remember how to do all of these things in your head. And also, it doesn't force you to learn object-oriented programming. I completely skipped that in the book. A lot of freshman computer science topics I entirely skip over. There's no sort of sorting algorithms or any discussion of things like that in the book. Python just allowed the code to be very terse. Kind of like Perl, except also readable, unlike Perl.

Who is Python for?

Joe: Yeah. So I agree. And what I love about it also is that even though Python is a programming experience in teaching beginners, you wouldn't consider it a toy language. You can use it for really advanced things?

Al: The software developer job I had for five years before I left to write the book full time was using Python and JavaScript. I mean, Python is up there up with C# sharp and Java. As far as professional software engineering languages, I think it's used more. Please don't quote me on this. I think Python is more popular in practical programming than Ruby even because it also has the Django web app framework similar to Ruby on rails. But, you know, Python can also be used as a scripting language. If you're a sysadmin and need to write simple scripts, or it can be used to write full-blown applications.

Joe: Awesome. So I don't know if you thought about this user case, but my audience is mostly test automation engineers, and some are coming from a non-developer background. What's great about this book is it teaches them the basics of knowledge of Python to get started with Python and get them up running quickly, and then they can apply those skills to test automation. So driving a browser.

Al: Oh, yeah. So the book is split into two parts. The first part is, is just a brand new python tutorial for people who have never programmed before. So it's fairly typical in that. But part two, which I think is about two thirds. The book, roughly, even if you're a seasoned developer, can just look at part two, which goes into, you know, there's a chapter on Excel, a chapter on modifying images, a chapter on GUI automation.

Selenium Automation and Web scraping with Python

Joe: Excellent. So what's the difference between web scraping and GUI automation?

Al: OK. So web scraping is I cover a few different modules for it. First, I start going off with requests and beautiful soup to download URLs and then parse the HTML from them. And that's nice if you have a simple static Web page where you need to find some bit of information that some buried somewhere in the HTML requests will help you download the Web page itself, and beautiful soup will help you parse it and find that information. But if you have something a bit more complicated, like it's a Web app, you need to fill out a form or log into something. The Web scraping chapter covers Selenium where which launches the Firefox browser that you can then simulate clicking on various form buttons and then filling in forms and interacting with JavaScript on the browser itself. However, for the GUI Automation chapter, this is just raw. You can control the keyboard and mouse and send mouse clicks anywhere on the screen or simulate typing. So this works with it could work with a web browser, but it could also work with any application that's on your computer. So if you have some program that really wasn't designed to be automated, if it doesn't have an API or anything like that, you can then just brute force, control the keyboard and mouse to have it automate all these tasks for you and just do all the clicking and typing for you.

Joe: So are there any examples in Automate the Boring Stuff with Python that you think most people either new to Python or even seasoned people that have been using Python script for a while aren't aware of? Maybe some functionality you think isn't used enough for people just don't know about it.

Al: I was actually surprised by how many modules there were for Python just for interacting with lots of different things like Excel files or passing PDF files. I was really afraid that a lot of this would have to sort of tells people to open binary files just from their Python program and try to figure out the format from there. So the second part goes into is basically a wide coverage of various third-party modules. So it makes it really easy to just I didn't realize that there was an Excel module that could open just like spreadsheets and then and not only just read and write cells to and from the spreadsheet, but you can also do things like styling and formatting and creating charts. And there's such a wide range of things that I didn't realize was even possible. I added the GUI Automation chapter at the very end just so as a sort of brute force way that, you know, if you want to do something and there is no module for whatever file format that you're working with, then if all else fails, you can just control the mouse and just have it click and operate the application for you.

Joe: I just want to reiterate that it's not hardcore automation, but it does allow you to automate some things that maybe the client application that you said doesn't have an API, but you still maybe could get around some simple things using it. I know Selenium, which was made for automated browsers, does have a Python language binding. So people, if once they learn Python could easily just import the Selenium binding and be able to automate a browser straight up in Python, which is pretty.

Al: Yeah, I always I had heard about Selenium before, but I always thought it was some hardcore automation tool. And then when I actually started using it for writing this chapter in the book, I was actually surprised at how fairly straightforward and easy it was to use python bindings. It makes it pretty simple. You can do most things in just a, you know, five or six lines of code.

Joe: I definitely agree. And what I love about it is especially with Python. It's just like you said; it's so much easier. It's not as verbose as something like Java or C# sharp. I just I'm always curious to know why more testers or developers don't use more of these scripting languages if they're doing automation.

Al: I would say that you know, a lot of people just learn some sort of software tool, and then that's sort of their main go-to for a lot of different things. I'm certainly guilty about that with Python. I think it's a great programming language, so I tend to use it all the time. But I remember thinking about Shell scripts. You know, I, I really have this huge distaste of Shell scripts, I have to say, because it's great when they're really small things like, oh, well, I want to run these five commands all at once so you can just put them in a shell script and that's fine. But then eventually, that script starts growing in size and you want it to do a lot more sophisticated functionality. And a lot of things are actually more suited for a real programming language. And I remember one time I was I was working with this one Shell script at work that somebody had written, and it was just this massive thing. And I thought I wonder if there's a. Bugger for Shell scripts, programming and that's when I realized maybe we should actually just convert this into an actual programming language and have access to all those tools. So I thought, you know, Python would be a pretty good scripting language for people to learn in order to do basically whatever they need to do and have the full power of Python in all of its source modules behind it.

Joe: I guess what I'm usually wary about is if I can use I use Java or C#. I know what I'm going to have an IDE. I'm going to have really good debugging tools. So if someone's starting with Python. Have you ever used something like Iron Python within Visual Studio or there are built-in debugging tools with just standard Python that someone could use?

Al: I haven't really. I mean, you know, it's it goes back to the hello world in Python is just one line long. I haven't found a need to have an IDE just to memorize all of the different python commands. It's usually relatively simple to do whatever I need to do in Python. I remember whenever I worked with Java and I have to open a file and read content out. I always have to look up the documentation for you know, buffer reader, and then connect to this and connect that. But, you know, with Python, it's just the open function. And then there's a reed method and, you know, and then I'm set. So I of mostly time, I'm just using something like Sublime Text or VIM or even sometimes just idle the IDE that comes with Python itself.

Joe: Is there one thing you see over and over again in your experience that most people either creating or learning Python are doing wrong?

Al: I have to say a lot of sort of new software, a lot of things that would be considered unsophisticated for a software engineer to do just like using lots of global variables or having something that's like an IF ELSE statement. That's several. You know, if else, if else, if else, if else if just incredibly long when it could just be tightened up. But for this book, I, I sort of am because I'm aiming it at complete beginners who are really interested in becoming software engineers. I really don't go into that. I think at one point I say, you know, it's fine if you just use lots of global variables because most of the scripts that are gonna be written by people using this book are probably only going to be a couple of hundred lines long at most. So so a lot of, you know, writing maintainable code techniques and things to remember. It would be nice if people learn that as well. But it's not really strictly necessary. And because Python itself is just so readable, to begin with, usually doesn't become a become a huge problem.

Joe: So I guess my next question was for someone that is using for their job, for development, at least developing test scripts. Any advice on how engineers can make their python code maintainable and less brittle?

Al: Well, there's actually there is Pep eight, which is the Python document that goes into just the style guide. And that gives a lot of really useful hints. I really also like the Google Python code style guide as far as how I just set up your code. The one place that I, I always disagree with PEP eight and people always e-mail me and say, well, you're violating PEP 8 here is I really prefer using snake case instead of the underscore is to separate words inside of variable names and other identifiers like that. It's always just been our preference of mine. But then whenever I produce any code whatsoever, I'm sure to have somebody come can say, well, technically, according to PEP 8, you should be adding, you know, underscores or something like. Yeah. And then I always point to the part of PEP 8 that says, you know. Just be reasonable, you know, a foolish consistency is the hobgoblin of small minds. Yeah, I mean, Pep eight says, you know, well, you don't always have to follow this document, but a bit more practically. I know Sublime text comes with an automatic Lintner that will check your code as you type it out. So if you're using Sublime Text or whatever editor you're using. You can have something like that which will just say, oh, you know, you forgot tangent, initialize this variable here or, you know, I've never seen this variable because you have a slight typo on that that can actually end up saving you a lot of time and going ahead. And I was sort of waffling a bit when I was writing this book about adding a chapter on debugging because I thought maybe that's a really hardcore topic that casual programmers wouldn't need to know. But you're learning how to use debugging and assertions and throwing exceptions, and those things can really save you a lot of time, even in the short term, I believe.

Joe: So you did mention at the beginning of the call that your, I guess, a full time writer now. So do you have any plans or what's in the works for any new books maybe where you do focus more on the developer type person?

Al: Oh, I like I keep telling my friends, like I think I've developed a book writing probably, um, because I wrote the first one, the second and third when I thought, OK, no more books. And then I thought, well, OK, I'll come out of retirement to pull off one last heist. I'm still interested in sort of programming education, especially for young people. So I had one idea for a scratched programming book and then also other ideas for sort of a freshman computer science text that we mostly used as much visualization as possible and had as little text as possible. There's the canonical introduction to Algorithms textbook, which I remember using four for one of my freshman computer science courses and thinking look like, wow, this is great. But it's also really dense tome and you really have to spend 20 minutes a page just parsing everything. And it would really be a lot nicer if this was had a lot more visualizations, a lot more graphs. And it's sort of, you know, explaining algorithms like I'm five years old. Yeah. I think that a lot of computer programming, it it has a sort of, you know, rocket surgery reputation of being incredibly difficult and hard when really that's not the case at all.

Joe: Before we go. Is there one piece of actual advice you can give someone to have them improve their learning Python and let us know the best way to find or contact you?

Also, check out my post on Best Python Tools for FullStack Automation Testing!

Actionable Python Advice

Al: The best advice I can give and this applies to complete beginners and also seasoned software developers, is that computers aren't magical. If you're getting some problem or something's going wrong, there ends up always being a reason. It might be this really obscure reason like your code was doing something that you really didn't intend or something that you didn't really think of before. But at the core of it, there is always gonna be some reason there. And so this is really helpful because a lot of times when I'm debugging, I just kind of like, well, maybe if I turn the computer off and on again, don't start working or, I don't know, just worrying that maybe it's, you know, I'm going crazy and this and that's the reason why this code doesn't work. But really, there's always ends up being some reason. It'll probably be something ridiculously simple that you've just forgotten. But there is you know, at the bottom of all of your searching for whatever the issue is, there will be some logical reason for it. Yeah. And I guess the best way to contact me is either email al at inventing with Python dot com or. Or just messaging any on Twitter where I'm at at Al Sweigart.