Who Grades the AI? 7 Questions to Ask Before You Trust AI Testing

AI is writing more application code than ever but verifying that software is still limited by QA capacity.

Teams evaluating AI testing tools face hard questions: What is actually covered? How much maintenance will this create? When a test fails, is it finding a product defect or is the test itself fragile?

At the same time, terms like agentic, autonomous, and self-healing can describe very different capabilities that look similar in a polished demo.

Anish Sharma, Co-founder of OttoTester, walks through seven questions testers, QA leaders, and developers should ask before trusting an AI testing tool and the evidence a credible answer should produce, not just a vendor’s “yes.”

Then he applies those same questions to OttoTester as a live example. You’ll see proposed coverage generated from a requirements document, the resulting test code, a healing record, and auditor scores backed by the evidence behind them.

What you’ll discover:

• The difference between AI assistants, test generators, testing platforms, and quality systems—and where each approach stops.
• Seven questions to ask before trusting AI testing, plus the evidence to look for in each answer.
• Two questions many evaluations miss: Does the system actually learn? And who checks the quality of its work?
• How to trace coverage back to requirements, so missing specifications surface before release instead of in production.
• What evidence looks like in practice: requirements-based coverage, generated test code, healing records, and 0–100 auditor scores with supporting evidence.

Learn more about OttoTester here.

About the Speaker  

Anish Sharma

Anish is co-founder of OttoTester. He has over thirty years in application architecture, development and testing and led engineering and QA teams for enterprise clients and federal agencies.