Testing the Unpredictable: Ensuring Quality in GenAI-Powered Applications
9:15 - 10:15 Great Hall 3
While AI-driven test automation has been widely discussed, the challenges of testing Generative AI applications are relatively new to most teams. Traditional QA methods struggle to address the dynamic and non-deterministic nature of AI-generated output. How do we define "correct" behavior for AI-powered features like chatbots, content generation, and recommendation engines, and how do we evaluate that in an automated way? What types of regressions are most likely to occur, and how do we detect them?
In this session, we will explore practical approaches to testing GenAI applications, from evaluating response consistency and accuracy to mitigating biases and unexpected failures. We’ll also discuss how the role of QA is shifting in an AI-driven landscape and what skills and strategies testers need to stay ahead. Attendees will leave with actionable insights to confidently tackle the quality challenges of this new AI era.
Dan has spent his career building products to make life easier for software teams. He is a co-founder at mabl, which provides a test automation product that uses AI to help teams deliver better software faster. Prior to mabl, Dan worked on the Cloud Platform team at Google, which he joined through its acquisition of Stackdriver, a company that he also-co-founded. Dan spent his early career in product and technical leadership roles at VMware, Sonian, and Microsoft. He lives in Boston with his wife and three children and enjoys golf, travel, and pickleball.