I’m really excited about this series. It’s the first time I’ve allowed a guest to post on my blog and it’s an honor to have Lisa Crispin as the first. For of those of you that don't know Lisa is a Test Advocate at Mabl as well as the co-author, with Janet Gregory, of More Agile Testing: Learning Journeys for the Whole Team, Agile Testing: A Practical Guide for Testers and Agile Teams, co-author with Tip House of Extreme Testing..
So let’s get into it. Take it away, Lisa.
The Automation Testing Struggle is Real
Software delivery teams have struggled to automate testing for as long as they’ve been delivering code.
We all know that software testing is an integral part of software development, along with coding and the many other activities needed to build valuable apps for our customers.
But somewhere along the line, there must have been programmers who decided they didn’t like testing, or maybe some managers who thought, “Let’s save some time here! Just crank out that code as fast as you can. Don’t worry about making sure it works. We’ll probably just have to rewrite it soon anyway.”
The truth about automation can be taught, and this is what you’ll discover in this three-part series, along with:
- You'll learn the value of automation, and what makes it such a big challenge for so many teams in their testing process.
- You'll see how visual models can help your team have the important conversations to build a strategy for test automation that works for your specific context.
- Lastly, you’ll explore the first category of models: the test automation pyramids and triangles.
FYI: To follow along with this recap you might want to also see mabl’s Modeling Your Automation Testing Strategy Part 1 webinar series by Lisa on this topic.
The Value of Automation
Automated regression tests give teams confidence that they can deploy or release small changes to production frequently. When every new commit can result in a release candidate, there’s no time for manual regression testing.
You might use feature toggling to keep a change turned off in production until you’ve had time to do activities that can’t be fully automated, such as exploratory testing. But if a change has broken existing capabilities, you want to know as soon as you can. Fast feedback loops are critical to being able to deploy quickly and reliably.
Automating the boring stuff frees up time for the more valuable testing activities including exploratory testing, accessibility testing, and security testing. You have a lot to learn about our product. Much of that learning depends on human observation and systems thinking.
Here’s the thing.
Automation can help us do many types of testing faster, like:
- Setting up test data
- Getting the app in a certain state for exploring
- Capturing visual changes to evaluate further
The fact is that automating tests as a team effort has benefits that go beyond the regression-checking safety net. In the process of collaborating to automate the right tests, team members in different roles build a shared understanding of how each system capability should behave.
In order to automate tests, they elicit rules and examples of system behavior from business stakeholders. They can turn these into executable tests that guide development and help the team know when a feature is done. And these tests become living documentation of how the system works, always up to date because the tests are running and passing multiple times per day!
Models are a tool
George Box famously said, “All models are wrong.”
He was talking about statistical models, and he probably wasn’t the first to think it. I like to think of models as a tool, not a rule. Get a cross-functional group of people from your team together. Draw a visual model on the whiteboard, gather around, and start talking about it.
Think about areas of high risk, of high value to the business. Do you spot any quick wins— something fairly valuable or risky for which you could easily think about how to create automated tests?
Every business and product is unique, every team has a unique mix of skill sets and experience, and customers come with many different needs and wants. I’ll be describing a wide range of models your teams might find useful to devise a workable strategy for automating tests in your particular context.
What Makes Model Based Testing Hard?
If you have a minute, stop reading and jot down some things that have gotten in your way when it comes to achieving your test automation goals.
When I have a big group of people do that, some common obstacles mentioned are:
- “We don’t have time right now.”
- “We don’t have anyone who knows how to do that.”
- “We can’t afford to buy (or learn) a test automation tool.”
- “Our UI changes so much, it doesn’t make sense to automate.”
Sometimes it boils down to fear, which is understandable. If you’re pressed for time, you lack the skills, you don’t have the right people, your application is unstable, or you don’t have any tools to help—you aren’t likely to make the significant investment that automating tests requires.
When you start an automation effort, you usually face a steep learning curve because:
- You have to learn basic principles, practices, and heuristics of effective test automation.
- You have to learn how to use a new tool or framework.
- You still have all the work of building the product features.
- The automation is extra work.
Brian Marick called this the “hump of pain”.
Eventually, the team builds up a library of reusable components, and gains expertise in the toolset. It also saves a lot of time because regression failures are caught quickly, and it’s saving a lot more time than it costs.
Detailed plans…or visuals?
You may have worked on teams where QA was tasked with providing voluminous test strategy documents. You may have noticed that few people, if any, actually read them. Try doing a simple one-page visual. You’re much more likely to engage the rest of the team with it.
More than once, I’ve gotten together with fellow testers or other team members to brainstorm our testing strategy on a whiteboard. We should then wheel it into planning and standup meetings and easily get the rest of the team talking about it and updating it.
A visual like this is easy to evolve as needed over time. It can be done virtually, too. Use anything that lets everyone in the group contribute—sticky notes on a wall, index cards on a table…jot down questions as you go. It’s the conversation that matters.
The Classic Test Automation Pyramid
Mike Cohn introduced me to his Test Automation Pyramid back in 2004 when he was my team’s manager. It worked really well for our team there, and I’ve shared it with many people who have found it effective to help them set goals around their automation testing and move towards them step by step. (To learn more check out Mike's book that covers this in more detail Succeeding with Agile.)
For some reason, people like to pick on the test automation pyramid now. They say “Oh, it’s really a triangle,” (yeah, so what?) and say it doesn’t work for everyone. Right—it doesn’t work for everyone—it’s a model. It’s a thinking tool!
Unit and Component Tests
The pyramid shape has unit-level tests and component tests (slightly bigger than unit level) as its strong base. These are the tests we can create quickly and maintain easily as we change the code. Generally, these tests are confined to the code. Calls to external components and to the database are usually faked, stubbed or mocked out.
The low-level tests run fast and give us the quickest feedback. Developers use them to ensure code correctness. In fact, they are used more for code design than for testing.
There are many great unit testing frameworks such as JUnit and TestNG available to make writing these tests easier. If we could test everything here, we would, but we can’t.
Note that we can do unit-level tests of our application’s UI as well.
The UI layer here in the pyramid is more indicative of tests that go through multiple layers, end-to-end through the whole system.
For tests that require us to go through more levels of code than the unit and component levels, we can move to the API or service layer. Here we bypass whatever user interface the product has and access API endpoints or service calls directly.
Many apps have a lot of business logic in this layer. Testing it without getting the UI involved saves a lot of test execution time. The tests are faster to create and easier to maintain, as a rule, than UI tests. They are likely to access the database and possibly external systems.
Successful API testing automation at this level requires collaboration between developers and testers. These tests promote collaboration and help everyone get on the same page.
You can use these tests to know when you're done with this mid-level business logic.
You also have lots of tools available to help you automate at this level, for example, check out the top API Testing Tools that are free.
UI Tests
Depending on how much logic and complexity are in the user interface, you’re going to have some number of workflow functional tests in that top layer of the triangle. You want to minimize these, as a rule, because they’re expensive to maintain and can give undependable test results.
They’re generally end-to-end tests going through all layers of the application—but they don’t have to be. You can also fake, stub and mock out components here wherever appropriate.
The more recent generations of test tools make it easier to create and maintain GUI tests, so the classic pyramid shape may no longer apply with those tools.
Test Automation Frameworks like open-source Jasmine help teams test drive UI code. There are countless libraries, drivers, frameworks, and tools today to help teams successfully automate at the UI level.
Be careful how you choose your tools. Just because you can create them quickly doesn’t necessarily make them easy to maintain.
Manual and Exploratory Tests
The happy cloud at the top of the triangle was the only way Janet and I could figure to represent the manual tests that shouldn’t be or can’t be automated, although automation can help speed them up.
Exploratory testing, accessibility testing, and security testing are just a few examples. We wanted to keep them visible as we think about automation.
Other Ways to Look at the Test Pyramid (or Triangle)
In his keynote at SwanseaCon 2018, Seb Rose presented a simpler triangle model which expresses the same idea. It’s also a great thinking tool.
It still shows our preference in quantity of tests for the fast, easily maintained tests that each exercise only a tiny part of the application, versus the potentially flaky, slower, expensive tests that give the whole application a workout at the top.
Aslak Hellesøy had another, if similar, perspective, which he described during his SeleniumConf keynote in 2018. He introduced the “Love/Hate Relationship” pyramid.
We love to write tests that take only minutes to write and run. We are kind of neutral about tests that take hours to write and run.
The tests at that top level could take days to write and get running, and require a lot of attention over time.
They are frustrating!
We don’t love them! But we need them. And, we can still find ways to speed them up.
Visualize Test With a Model
Here’s your challenge. Choose a feature or story that involves different layers of your application. Draw a test automation pyramid on a wall, paper, virtual whiteboard or whatever works. Identify the tests you would do for this feature or story.
Think about the risks posed by different parts of the application. Discuss what matters to the customers, what is valuable to the business. Where in the model would these different tests fit?
As your team looks at this visible representation of automation, you may come up with new ideas or implementations of your features to make them easier to test at whichever level is appropriate.
The visual-driven testing model also helps you prioritize areas where you want to automate first.
Next time
In future posts, we’ll explore more models and visualization techniques to help your team discuss and form a successful automation strategy for your test scripting efforts.
And at some point, we will start thinking about how to put the automation strategy of our test cases into action!
Also as I mentioned earlier you follow along with this recap by viewing mabl’s Modeling Your Automation Testing Strategy Part 1 webinar series by Lisa on this topic.
