Lee Barnes
Chief Quality Officer, Forte Group
About
I have over 30 years of experience in the Software Quality Engineering field. I have been involved in the implementation of test automation and performance engineering solutions in hundreds of environments across a wide array of industries. Most recently, I’ve been involved with augmenting Quality Engineering processes with AI. I regularly speak at industry conferences including QA or the Highway, DevOpsDays, STAREast, Targeting Quality, and Innovate QA
Taming the Beast: Testing Non-Deterministic AI Systems with Confidence
Description
For decades, software testing has relied on a comforting assumption: given the same input, systems should produce the same output. AI-enabled systems break that assumption entirely. Large language models and other AI components generate responses that can vary in structure, tone, and content while still appearing “correct”.
In this session, we’ll explore why traditional testing strategies struggle with non-deterministic AI behavior and where they quietly fail. Using real-world examples such as AI chatbots and resume-screening systems, we’ll walk through practical techniques for validating AI outputs without relying on brittle, deterministic assertions. Topics include input variation strategies, semantic similarity analysis, bias detection, and using LLMs responsibly as automated evaluators (aka “LLM-as-a-Judge”).
Attendees will leave with a clear mental model for testing AI-based systems, concrete patterns they can apply immediately, and guidance on balancing automation, human judgment, and risk. If you’re responsible for the quality of AI-driven features, this talk will help you move from uncertainty to confidence!