Evaluating What Matters: Using Qualitative Evidence (and AI) to Prove Learning Impact
Learning teams are being asked to prove impact, yet one of the richest—and most underutilized—sources of evaluation evidence is qualitative data: learner reflections, open-text survey responses, focus groups, peer feedback, and manager observations. Too often, these inputs are treated as “anecdotal,” summarized loosely, or ignored altogether because teams lack time, tools, or a repeatable method to analyze them. The result is a familiar gap: dashboards show activity and satisfaction (completions, NPS, ratings), but they rarely explain why a program worked, what changed in behavior, or what should be improved next.
In this interactive session, participants will learn a practical workflow to convert qualitative data into credible impact evidence and decision-ready recommendations. You will practice a lightweight coding approach to surface patterns (drivers, barriers, enablers, and behavior-change indicators), then connect those themes to quantitative measures and business outcomes to strengthen conclusions. Participants will also learn how to use generative AI responsibly to accelerate qualitative analysis—such as first-pass theme extraction, clustering, and draft synthesis—while applying human validation steps to reduce bias, protect confidentiality, and ensure defensible findings.
Through case examples and hands-on activities, participants will leave with tools and templates to elevate learner voice into actionable impact insights—what to continue, scale, redesign, or stop—supported by evidence leaders can trust.
Learning Outcomes (measurable)
By the end of this session, participants will be able to:
- Explain why qualitative evidence is frequently underutilized in impact evaluation and how it strengthens causal interpretation (the “why” behind results).
- Select high-value qualitative sources and design prompts that produce evaluation-ready evidence (not vague opinions).
- Apply a streamlined coding/theming method to identify drivers, barriers, and indicators of behavior change.
- Use generative AI for rapid first-pass qualitative analysis (theme extraction, clustering, synthesis) and validateoutputs using a human-in-the-loop checklist.
- Integrate qualitative themes with quantitative measures to draft a one-page, decision-ready Impact Brief with clear recommendations.
Practical Takeaways (what participants leave with)
- The repeatable workflow: Collect → Code → Connect → Communicate
- A prompt bank for open-text surveys, reflections, and focus groups
- A “human-in-the-loop” validation checklist for AI-assisted analysis (accuracy, bias, confidentiality, traceability)
- An executive-ready Impact Brief template (findings → evidence → implications → recommendations)
.jpeg)
Christopher Massaro is a learning and organizational development professional with more than 25 years of experience designing, facilitating, and evaluating training programs that support leadership effectiveness, organizational performance, and learner growth.
He serves as a Senior Talent Development Consultant at UNC Health, where he supports leadership development, organizational development, competency-based learning, and enterprise learning initiatives. His work focuses on helping leaders and teams translate learning into practical behavior change, stronger performance, and measurable impact.
Christopher also serves as adjunct faculty at the University of Southern Maine, where he has taught leadership studies and social science research methods. This background informs his practical approach to learning analytics, evaluation, and evidence-based program design.
ATD Capability Model:
Personal: Collaboration & Leadership, Lifelong Learning
Professional: Evaluating Impact
Organizational: Data & Analytics
.png)