Beautiful Property-Based Testing (Falling in Love with Testing)

If there’s one thing I know for sure about testing, it’s that there is never enough time to write all the tests I’d like to.

Even if you create lots of tests, it doesn’t mean you are actually covering all aspects of your application.

Good tests are difficult to write, and even when they are good, we sometimes miss critical edge conditions with that testing.

So what to do?

Property-Based Testing to the Rescue

One answer–aside from unit testing and slow, fragile UI functional automated testing, is to use property-based testing.

Before we look at some property-based testing tools, let’s first make sure we’re all talking about the same thing.

What is Property-Based testing?

I’m not an expert on property-based testing, but I recently spoke with a few developers on Test Talks who are familiar with property-based testing. Here is how they described it to me.

Eric Normand of PurelyFunctional.tv defines property-based testing as a kind of random testing. It basically means you generate random data.

For example, instead of testing for a known, deterministic value you can instead send random inputs to your functions.

You don't need to know what the result is supposed to be beforehand. You may only know, for instance, a mathematical property such as the value will always be greater than zero.

If you could come up with a whole bunch of similar properties, you could actually express a whole class of tests.

This is pretty cool, because instead of having just one or two examples you can have the computer generate hundreds or even thousands of tests that randomly finds bugs in your code.

The Beauty of Property-Based testing

Scott Nimrod says the beautiful thing about property-based testing is that you're letting the framework manage the edge cases for you.

That is indeed a beautiful thing.

It complements your unit tests, in which you leverage your property-based tests to catch the edge or special cases. This is also something that application developers can easily adopt or take into their arsenal of tools.

Better Option than UI Automated Tests

I hate it when the only automation strategy a team uses is UI-based automation. UI automated tests are brittle, hard to maintain and often unreliable. This is due to a host of issues which I’ve listed in my Automated Testing Best Practices post.

So whenever I hear of any testing techniques where you can use automation without relying on a UI, it makes my heart skip and puts butterflies in my stomach.

I about lost my mind when Eric gave a beautiful example of this. The company Pivotal Tracker was able to use property-based testing for their application. But instead of going through the UI and clicking buttons, they actually found the functions those buttons were bound to.

Based on this they could actually generate random calls to their back end because they knew the functions the buttons would have triggered. With this method they were able to test everything in their program without having to run end-to-end UI tests.

Pretty sexy, right?

Get Started with Property Based Testing

Now that I’ve gotten you revved up, you’re probably wondering how you can get started with property-based testing.

Eric offered some great actionable advice about this. He recommended finding a property-based testing library in your language, then convert some of your example-based tests. Use random data instead of the examples you've hard coded in in your test. Simple.

Scott Nimrod had an awesome session at this years Automation Guild on Property-based testing with a hands on coding example on getting started using F#.

Property-Based Testing Libraries

The good news is that there is most likely a property-based library you can leverage now to help you get started.

Here are some of the most popular property-based testing libraries I could find for most programming languages:

• Java – junit-quickcheck is a library that supports writing and running property-based tests in JUnit, inspired by QuickCheck for Haskell. Checkout Bas Dijkstra's post on an introduction to junit-quickcheck.
• Javascript – JSVerify is a property-based testing library, highly inspired by QuickCheck. It is testing framework agnostic, you could use JSVerify with Mocha, nodeunit, Jasmine or any other framework. Check out an introduction to JS Verify
• .NET – FsCheck is a tool for testing .NET programs automatically. Check out introduction to FsCheck example
• Python – hypothesis-python Advanced property-based (QuickCheck-like) testing for Python. For code example check out Elliot Chance's Using Hypothesis in Python
• Ruby – Rantly is a Random Data Generator and Quickcheck. You can use Rantly to generate random test data, and use its Test::Unit extension for property-based testing. To get started with Rantly checkout Sitepoint's post on Property-Based Testing in Ruby

Good luck, and enjoy falling in love with testing all over again.