I’ve been testing since the mid-’90s, but automation has gotten more and more complicated as the years go by. Although black-box testing has always been difficult, it has become even harder at the Enterprise level.
This is mostly due to end-to-end workflows that involve package products like SAP, Salesforce, Oracle, etc.
So black box testing is an important phase of the software testing process where the tester does not see the code or content of the software.
If you're wondering what the term “black box” 📦 means, it refers to a lack of information regarding the internal operations of the program, or of anything else within the box.
So how should you balance the complexity of automation of business processes at scale with the need for today’s Enterprise type of testing?
Read to discover:
-
- The Danger of Ignoring Testing Packaged Applications
- Integration Testing Critical for Black Box Applications
- What is Model-Based Testing?
- Testing all Layers of an Application
- Most Automation Tools Don’t Automate Black Box Applications
- Eggplant DAI Black Box Case Study
- Model Making in Eggplant (Actions and States)
- AI-Driven Black Box Exploratory Testing
- Automatically Increase Test Coverage
Why Do Testers at Large Companies have to deal with Black Box Testing?
Ethan Chung from Keysight Technologies told me that most large companies, no matter how big they are, still have their own in-house development environment.
With most enterprise environments now embracing a hybrid approach with both on-premises licensed software and software being SaaS (software as a service), and many applications and workloads moving to the Cloud, the environments become more challenging as well.
Most teams no longer own and control all the code that goes into their applications since more and more in-house software is being integrated with the off-the-shelf products/package solutions like Salesforce, SAP, Oracle.
Quality Assurance testing now has become less a matter of “how do we test, what is our code?” than “how do we test things that have evolved way, way beyond the scope of what we can manage?”
Also, many companies use extensions on top of package solutions like Salesforce. That adds complexity since the app represents layers and layers of code that testing teams don’t have access to.
Trying to automate is difficult because these apps are truly pure, black-box testing environments.
The Danger of Ignoring Testing Packaged Applications
For all the aforementioned reasons, many applications are completely ignored since most teams have the “not my code, not my problem” mentality toward them.
Needless to say, this is a major issue.
Even worse—most package-based applications tend to be the most mission-critical for a business.
If a company is using Salesforce, an SAP Application, or anything Oracle they are so embedded within the organization's infrastructure that if it goes down, productivity stops.
Bringing in outside consultants won’t help much either because they’ll only test their own piece of infrastructure.
For example, SAP is never going to be testing Oracle software. Right?
Integration Testing Critical for Black Box Applications
So how do you make sure you have a black box testing approach that could fit across all these items? Because it's generally the integration that causes problems.
Integration is usually the issue because all these systems use fundamentally different data schemes.
You need to test how the data looks and behaves crossing between the different systems.
This is difficult to spot because Salesforce can intake data and it's technically correct, but you don't have a clue whether it's correct or not until a user gets on it and it hits production.
To solve this Ethan has seen the model-based approaches to automated testing work extremely well for enterprise-level black box testing phases.
What is Model-Based Testing?
What is a model?
According to Webster's dictionary, there are multiple definitions. It can be a miniature representation. An imitation or emulation. It's something that can't be directly observed that we visualize. It could be a computer simulation.
Testing expert Jim Hazen mentioned in his Automation Guild session on Model-Based Testing that a model is a physical representation of a thing or system for emulation and visualization.
Systems modeling is the use of a model or models to conceptualize a business system or something else in the development itself.
How Does this Apply to Real-World Black Box Testing?
Most companies have their own custom-developed front end, which uses behind-the-scenes, off-the-shelf backends, and databases.
So, using web-based Selenium functional tests will not get you the coverage you need.
Testing all Layers of an Application
For these environments, it’s critical to have one testing solution that can orchestrate the business requirements of the test across all of these layers.
For example, with a model-based approach, you have one section of the model that focuses purely on the browser, which is one tech stack and how the user can interact with it.
The second a user enters an order from the browser UI, the test can jump straight into your Salesforce environment.
This allows you, for example, to verify that a purchase order just entered by the browser is the correct one. And the data entered matches the actual items added. So you would do a business logic validation there. You would typically do other validations as well, ensuring that the currency is correct, the actual numbers are correct, and the order actually gets fulfilled.
Using model-based approaches will find the integration issues across different environments while avoiding the data issue we talked about earlier. It also helps test the critical boundary values of your application.
Most Automation Tools Don’t Automate Black Box Applications
That’s why using a test automation tool like Eggplant, which handles all kinds of technology. really shines in these scenarios. Especially with what has traditionally been called non-functional testing.
Many automation solutions rely on being able to see and interact with the elements under test, so they tend not to be great choices for package-based, black-box test automation.
Eggplant handles it using a sophisticated image-recognition, OCR approach.
So if you can get a user to connect to the systems, you can go from a browser perspective on the actual website straight to your Salesforce, using the exact same test that can run the queries via the database tooling.
This gives you the ability to validate how the full flow works.
Automating the application like a user would “visually” on these black-box environments, your test is accessing your application as if it is through the eyes of a real user. Without a visual approach, you’ll always have gaps inside any complex system.
After speaking with Ethan on my podcast I decided to give it a look.
Eggplant DAI Black Box Case Study
Eggplant uses DAI, which stands for Digital Automation Intelligence, and it's a combination of artificial intelligence, machine learning as well as their automation platform that performs complex, E2E, black-box testing.
The first thing I noticed was that all the types of testing activities can be performed and viewed in one dashboard location.
This makes it easier for the whole team to see what’s going on.
As I mentioned earlier, Ethan recommended a model-based technique for automation, and Eggplant follows this approach.
Eggplant already has built-in, existing models for different applications and systems.
It also gives you the capability of building a model of exactly what you want it to be; you can scale a model according to your development team's specific needs.
For example, if you’ve got one team developing one side of an application, and another team developing another side, you could have two separate models.
These could then be combined as two sub-models to a greater model.
The key emphasis here is that the granularity and configurability is in your hands; you aren’t locked into having to specifically create a model for the whole of the application or just a single portion. You can do it however you wish.
Model Making in Eggplant (Actions and States)
When creating a model of your software in the Eggplant platform, you’re creating a digital twin of your application system.
There are two concepts you need to know when developing a model.
The first concept is the gray boxes that Eggplant refers to as actions. Actions are what a user would typically interact with, like a sign-in, submit, password, and so forth.
The second concept is known as states. You can think of them as pages if you like. States encompass your actions, i.e. returning customer sign in your details and constantly and so forth.
By having this model and this high-level overview of your application system, you can see the real behavior that a user would exhibit against your application system.
For example, if you have an action for “my account,” the automation behind Eggplant wouldn't go from “my account” down to “my orders,” because it knows an actual human user wouldn't be able to do that because he’d have to sift through several flows of working.
This ensures you have a model that exhibits the behavior you're testing against is only the behavior a user can do, and only performs actions that a real user would perform.
Once you have a model, Eggplant can then leverage its AI and machine learning algorithm to do cool things like exploratory testing.
AI-Driven Black Box Exploratory Testing
Eggplant can now use your model and its machine-learning algorithm to crawl through and test all the intricacies of your application. This means that your Eggplant instance will essentially appear to do what it wants to and start proceeding through the different models and the different states of those models.
Your testers can also leverage the Directed Test Case by using the built-in Test Case Builder.
This makes it super easy to use to build a new regression testing test case.
This means groups that may not have programming knowledge can still create the test cases they need—and which are critical to the business—without having to code it themselves.
It’s the best of both worlds; you can have human business experts leverage a tool that does the code development for them.
Automatically Increase Test Coverage
Another time saver this model-based, behavioral testing approach provides is the fact that you can test uniquely against end-to-end journeys, with multiple applications on multiple platforms and even multiple operating systems.
For example, I saw a demo of Eggplant where they showed a script that jumped from an iOS mobile appliance to Windows OS, back to the mobile device to get a two-way authentication code, then back to the website to add the authentication input values:
Using a vendor tool like the Eggplant DAI platform lets you jump between different software applications and systems to take actions live as they're happening, and feed the info back into the tests at runtime with no human interaction needed.
This is done using AI, machine learning, and optical character recognition (OCR).
By using OCR as well as image recognition, it gives the tool the ability to solve the challenge of running the same tests against different configurations and devices like iOS and Android—all with different screen sizes.
This is just a small sample of the types of software testing vendor-based, black-box testing tools can do for enterprise-wide, end-to-end testing.
