Learning Evaluation as a Competitive Advantage with Will Thalheimer

Leading Learning Podcast interviewee Will Thalheimer — Leading Learning Podcast interviewee
Will Thalheimer

Decades of research say the traditional learner survey—the smile sheet many learning businesses still rely on—is virtually uncorrelated with whether people actually learned anything. So what would it take to do better?

In this episode of the Leading Learning Podcast, co-host Celisa Steele talks with Dr. Will Thalheimer, author of Performance-Focused Learner Surveys and someone wise in the ways of science-backed learning evaluation. Will shares takeaways from his global survey of how workplace learning professionals evaluate their programs, the three fundamental reasons to evaluate learning, and how his LEADS framework (Learning Evaluation As Decision Support) can help a learning business decide which programs to keep, revise, retire, or reprice.

To tune in, listen below. To make sure you catch all future episodes, be sure to subscribe on Apple Podcasts, Spotify, or wherever you listen to podcasts.

Listen to the Show

Learning Evaluation as a Competitive Advantage with Will Thalheimer

Apple Podcasts Spotify Leave a Review

Access the Transcript

Download a PDF transcript of this episode’s audio.

Read the Show Notes

Celisa Steele: [00:00:03] If you want to grow the reach, revenue, and impact of your learning business, you’re in the right place. I’m Celisa Steele.

Jeff Cobb: [00:00:10] I’m Jeff Cobb, and this is the Leading Learning Podcast.

Celisa Steele: [00:00:17] As our guest for this episode notes, hardly anyone gets fired for doing bad learning evaluation. But there are good reasons to try not to do it badly.

Jeff Cobb: [00:00:26] That guest is Dr. Will Thalheimer, a consultant, speaker, researcher, and founder of Work-Learning Research. Will has spent years studying learning evaluation and translating it into practical guidance and even developing the Learning-Transfer Evaluation Model (LTEM).

Celisa Steele: [00:00:42] He’s written the books Performance-Focused Learner Surveys and The CEO’s Guide to Training, eLearning & Work. And, earlier this year, Will completed a comprehensive study on how workplace learning professionals evaluate learning, and we talk about some of the takeaways from that survey.

Jeff Cobb: [00:00:59] You and Will also talk about three major reasons to evaluate learning and the value of using evaluation to support the decisions a learning business has to make about its portfolio.

Celisa Steele: [00:01:11] Will also makes the case that good evaluation is a potential competitive advantage for learning businesses.

Jeff Cobb: [00:01:16] If you want to make sure your evaluation practices are helping you improve and not just checking a box, we think you’ll get a lot from this conversation with Will Thalheimer.

A Few Takeaways from Global Research on Learning Evaluation

Celisa Steele: [00:01:33] Earlier this year, you released results from your global research on workplace learning evaluation, which is a survey of hundreds of L&D professionals around the world about how they evaluate learning. Most of your respondents work in a corporate setting, but about a quarter serve learners outside their own organization, and about another fifth serve both learners inside and outside their organization. I would love to hear from you what surprised you most in those results, and, in particular, which findings do you think might be most salient for learning businesses—those listeners that we have for the Leading Learning Podcast?

Dr. Will Thalheimer: [00:02:13] A couple of things jumped out at me. Number one, learning professionals are still frustrated with their ability to do the learning evaluation they want to do—that’s across the globe, across industries, across the kind of industry, the kind of organization you’re with, et cetera. The other thing that jumps out is we are primarily relying on learner surveys. Even when we ask people, “How are you measuring impact?,” the number one thing people say is that they are using learner surveys. And that’s not good because a survey is not the best way to find out if people are actually having an impact. Another one of the findings: When we look at those learner surveys, one of the things I did was I gave people 20 different learner survey tactics, ways you could do them. Whether you are using Likert scales, numeric scales. What kind of questions are you using? Are you using sliders, hearts, stars, or lucky clovers? What are you doing? And a lot of those are not best practices, but we’re doing a lot of those. There’s a lot of room for improvement in general, and that’s an opportunity for all of us in the field. Because, if the baseline is not very good, and we do better than the baseline, then we’re creating advantages for ourselves, our learners, our organizations.

Dr. Will Thalheimer: [00:03:45] I wouldn’t say this is a surprise because this is my hypothesis, but one of the interesting questions we asked was, “What are senior leaders telling us they want in learning evaluation?” And my hypothesis, from being in the field for so many years, is that a lot of them aren’t paying attention. A lot of these senior leaders aren’t really paying attention to what we’re doing in learning evaluation, and we—L&D folks—don’t always have a one-way connection with them. I ask people, first of all, “Hey, you’re L&D folks. Have you talked to your senior leadership about learning evaluation?” Only 31 percent said yes. A lot of people said no. And some people said, “I’m pretty sure that our leadership doesn’t care or doesn’t have an opinion about learning evaluation.” So then I took the subset of people that said, “Yes, we talked to our leadership about learning evaluation.” “Great. What do they want?” Somewhere around 30 percent said, “We want to measure our work performance and work results.” But, if you multiply the 31 percent by the 30 percent, that’s a pretty low level. It’s like 11 percent or something like that—not that many. And the second highest grouping of those who talked to their learning leaders said that the learning leaders have no opinion, or they said, “I’ve talked to them, but I’m not really sure what they’re thinking about it.” What’s interesting about this is that there have been a lot of evaluation gurus over the years telling us we must measure impact because that’s the most important thing to senior leadership. Also, we in L&D are going to get fired if you don’t do this. And that’s one of the things we found—that’s one of the lowest problems if you don’t do learning evaluation. Hardly anybody gets fired for doing bad learning evaluation. A bunch of different stuff. This is a long report. It’s over 100 pages, so there’s a lot in there. But those are some of the highlights.

Balancing Measures That Predict Repeat Business Versus Measures of Learning Effectiveness

Celisa Steele: [00:06:02] When I think about learning businesses, many of them track satisfaction scores, so they might be using something like the net promoter score. But you and I know that those measures say not a lot about whether people actually learned. I would love to hear from you on how a learning business should be thinking about balancing measures that can predict whether those customers will come back and enroll in another course/pay for another course with measures that tell them whether that learning experience, that course, say, actually worked.

Dr. Will Thalheimer: [00:06:38] Those of us who sell learning obviously have to talk to our customers and see what resonates with them. Sometimes, if they tell us, “Hey, I want the net promoter score,” well, we probably need to use that. But most of us are in the field because we believe in what we’re teaching, and we want to do that well. I like to talk about three major reasons that you might want to measure learning. One is to demonstrate the value—prove that you’re doing well. Another is to support our learners in learning—we can do some measurement stuff that directly helps our learners. And the third reason is to improve our learning—keep what’s good. It’s more complicated than that, but, if you think about those three things, if you are improving your learning, creating those cycles of improvement, you’re going to do better to support your learners. You’re also going to create more value, and then it’s a lot easier to demonstrate that value. So, to me, the fundamental is improving.

Dr. Will Thalheimer: [00:07:44] Let’s just focus on learner surveys right now. We ask questions about the reputation of the course, like the net promoter score or general satisfaction—we’re learning a little bit. But traditional smile sheets have always done that, and there’s a whole bunch of science on this. There are over 150 scientific studies and two meta-analyses that cover these things that show that those kinds of questions are uncorrelated with learning results. So you get high marks on your net promoter score and have a good course but, equally likely, a bad course. You get low marks; you could have a poorly designed course but, equally likely, a well-designed course. With some of our traditional smile sheet questions, we’re going to leave some blind spots for ourselves. My argument is can we do better? Yes, we can. We can ask questions that are more aligned with what we know about learning. If we ask a question, “Hey, are you going to be able to put this into practice in your work?,” it’s focusing on their performance.

Dr. Will Thalheimer: [00:08:51] We ask them, “Are you going to be motivated to apply what you’ve learned?” That’s one of the keys. There’s this research on transfer, whether people will use the information in their work, and there are several key things in that. One is motivation to apply. Another is did they understand the concepts in the first place? If you don’t have a clear idea, then you’re not going to be able to apply them very well. Another aspect is remembering. Are you likely to remember what you’ve learned? Because, if you don’t remember it, you’re not going to use it in your work. Now, you can’t ask people directly, “Did you remember this?” People’s minds are set up to say yes to that question because everything’s top of mind when you ask them. But you can ask them, “Could you get the kind of interactions, did you have the kind of experience that would set you up for remembering? Did you get a lot of rigorous practice? Did you have realistic discussions? Did you figure out different contingencies or ways to do things?”

Dr. Will Thalheimer: [00:09:54] We ask that, then we’re more likely to know about remembering. I call these the four pillars of training effectiveness. The final one of these four is are you getting after-training support or after-learning support? Do you have a coach to mentor you? Do you have job aids to use? Are you going to be reminded of what you learned? We know that after-training support is valuable for people to be able to apply what they’ve learned, so let’s use questions that are focused on these things that are related to learning effectiveness. Also, let’s ask better questions. Likert scales/numeric scales are not good enough. They’re too fuzzy. People have a hard time answering them. In the old days—you’ve seen the paper-based smile sheets—people would circle the same number down the whole smile sheet. That means they’re not paying attention. That means they’re not really answering your questions. So, if they don’t answer your questions, if they don’t think about them, you’re getting bad data. And, if you’re going to make decisions based on your bad data, well, that’s no good.

Why Measuring Impact Isn’t Enough and the Potential for Evaluation to Be the Source of a Competitive Advantage

Celisa Steele: [00:10:59] We’ve already been talking about impact some, and I know that you have a paper called “Ten Reasons NOT to Measure Impact.” You’re arguing that impact really shouldn’t be a top priority in learning evaluation. But we also know many learning businesses feel pressure to show impact or at least a return-on-investment-type value. Even though it can be a real challenge to potentially assess that impact, especially once learners are back on the job and out of the course and maybe away from the learning business that has provided this experience, what do you recommend? What do you think? What’s realistic? What can a learning business do when they’re pressed to demonstrate impact? What would that look like?

Dr. Will Thalheimer: [00:11:43] Let me start with that article because I wrote a provocative title to get eyeballs. I’m sure your listeners can relate to that. My point was not to not measure impact—my point was don’t just measure impact. Because, if you just measure impact, then you don’t know all the learning factors that are going to enable you to improve your course. If you just measure your ROI or your KPIs, then do you know whether your learners are likely to remember? Do you know if they understood the content in the first place? Do you know if they’re motivated to apply? Do you know if they’re getting after-training support? Do you know if they can make better decisions? Do you know if they can take better actions? If you don’t know all those things…those are critical for us to create those continuous cycles of improvement. So it’s not that we shouldn’t measure impact, but we should not just do that by itself.

Celisa Steele: [00:12:43] There’s also, I think, a bit of an argument too around how that can be tricky if you’re going to measure it well. You were talking about, even with impact, a lot of your survey respondents were saying they were still relying on learner surveys to determine impact.

Dr. Will Thalheimer: [00:13:00] There’s more you can do. A lot of us use knowledge checks. Now, a lot of our knowledge checks aren’t that well designed. They tend to ask questions about trivial knowledge versus foundational knowledge—in other words, knowledge that helps people make decisions. But that’s probably a little bit better. We can also use my favorite thing, one of the best investments, high value, is using some kind of scenario. I’ve taught courses on how to write scenario questions. I’m a big believer in this. I used to work for a company that built simulations based on scenario questions. But they’re one of the most powerful things we can do—give people a realistic scenario, and then write these questions; get them vetted by experts. There are ways to validate your questions so that they differentiate between those who know—who have some expertise—versus those who don’t know. There’s a whole process behind that. But that kind of focus on decision-making competence is something that’s valuable. Let me back up and say those of us who are in the learning business, or the business of selling learning, have competitors. And, if everybody is using a traditional smile sheet, if we’re using a traditional smile sheet, then we’re doing what our competitors are doing. We don’t have an advantage over our competitors. We’re doing the least we need to do. How can we gain a competitive advantage? We need to do more than that. One way is to create learning that’s more effective.

Dr. Will Thalheimer: [00:14:45] And, if we have better evaluation, we’re going to create those cycles of learning design improvement. But we can also go to the marketplace and say, “Hey, we used to just measure smile sheets. We used to just ask the learners. But now what we’re doing is this—we’re using scenario-based questions. We’ve done a pre-test to post-test.” I always recommend to organizations, if they can do it—it’s not always easy—to have trusted partnerships with some of their clients. Bring them in and say, “Hey, we want to do some evaluation with you, and we’re not going to charge you for this, or we’re not going to charge you very much. We just want to see what happens and do some comparison testing. And the benefit to you will be you’ll get really good information about the program.” We’ll be improving our programs, and then we can create some content marketing around this. Instead of just going out and saying, “Oh, our course is great; we have good content; yada, yada,” we can say, “Hey, we were doing some research on this, and this is what we found. And we, as an organization, are dedicated to improving what we’re doing and making sure it’s world-class.” That’s a big opportunity here to think, number one, can we create a competitive advantage? Number two, can we create content marketing that enables us to differentiate ourselves from our competitors?

Celisa Steele: [00:16:16] I really like that idea of thinking about evaluation as a potential factor on which to differentiate from competitors. That’s a really interesting takeaway for listeners to think about that lever, if they’re not already thinking about that, as something they could be playing with to improve their offering.

Using Learning Evaluation Data to Make Decisions

Celisa Steele: [00:16:34] You’ve written a lot about learner evaluation, but you’ve also written a book called The CEO’s Guide to Training, eLearning & Work. I want to talk about a framework that you have in that book called LEADS, which stands for Learning Evaluation as Decision Support. What does it mean to treat evaluation as a support for decisions rather than proof of value? And how should learning business leaders make decisions with evaluation data? I’m thinking about things like which programs to keep, retire, or revise, but maybe there are other decisions that you have in mind as well.

Dr. Will Thalheimer: [00:17:14] Five, ten years ago, I was thinking, “Why do we do evaluation in the first place?” And I thought one of the key things is we want to help ourselves make better decisions. We want to get data that helps us make better decisions so we can take better actions. Well, that’s interesting. If our end goal is better decision-making, why don’t we start with what decisions we have to make and then build our learning evaluation system to help us make those decisions? So then I started writing about this, and I wrote a couple chapters in that book about this idea. You can do this in a super complex way, and I give people some ammunition in the book to do that. Or you could do it in a simple way—every time you’re thinking about what you’re doing for evaluation, ask yourself, “What decision is this going to help me make?” Even if you’re just going to use a survey, for every question that you are contemplating asking, ask yourself, “What decision will this question enable me to make?” And, if your answer is, “I don’t know,” then don’t ask that question.

Dr. Will Thalheimer: [00:18:27] In the book, I offer 14 categories of decisions that people in the learning field have to make. Those are things like how do we support remembering? How do we support understanding? How do we support motivation? How do we make ourselves credible to our business leaders? You can decide what you’re focused on and then work backwards. You mentioned things that I haven’t thought a lot about—what course to keep and what new course to deliver. The one caution I have is sometimes we ask people questions, and they may not be able to tell us whether it’s really going to work for them. Back in business school, they told us, “Don’t just survey people. If you want to know whether they’re going to buy a product, you create an MVP; you create something.” In the workshop world, we could do something like, “In six months, we’re going to offer this course, and, if you sign up now with our early-bird discount three months in advance….” And then, if nobody signs up for your thing, then you know maybe this is not something we should build out.

The Potential for AI in Evaluating Learning

Celisa Steele: [00:19:52] I feel like we should talk about AI. It’s the kind of thing you almost have to do these days.

Dr. Will Thalheimer: [00:19:58] Yes, that’s right.

Celisa Steele: [00:20:00] You asked a little bit about AI in that survey that we were talking about, and you found that AI is near the top of the list of what people are hoping to try in evaluation, but you were cautioning that it could be herd behavior. We’re all talking about AI, we’re all hearing about AI, and so maybe it’s more herd behavior than being driven by a strategic choice. Talk about where and how you think AI can contribute positively to how a learning business handles evaluation, both in terms of potentially creating evaluations and then analyzing the results of those evaluations, the data that comes from them.

Dr. Will Thalheimer: [00:20:43] So this GROWLE [Global Research on Workplace Learning Evaluation] research was conducted in January and February and a little bit of March this year. It’s very recent, but AI is moving fast. We’re now in the September-October area, so things could have changed. But what we found earlier this year was that about half of workplace learning professionals’ organizations were already thinking about using AI for learning evaluation, but half weren’t. Also, hardly any of them had vendors who were doing learning evaluations. So, if anybody’s out there, that might be an opportunity for somebody to do that. But what was interesting to me is we asked people, “What are you doing now in using AI, and what might you want to do next year?,” and the kind of things people were doing were fairly straightforward, solving some big problems in evaluation. One problem on a survey is open-ended responses, comment questions. In the past, we’d have to sift through those with our eyeballs, and we might—if we were sophisticated—do category sorts and things like that. Now AI can do that very quickly for us, and now our open-ended questions can have a lot more benefit for us. That was one of the use cases that everybody’s doing.

Dr. Will Thalheimer: [00:22:10] People were also using it to analyze data. We asked one question: “What learning-evaluation innovation are you hoping for in the next couple of years?” AI was the number one thing, by far. Then we asked them, “What do you think you want to do with AI and evaluation?” And the responses were very generic—most of them. Certainly the things I’ve already mentioned—the open-ended responding, processing that, data analysis—were all talked about. But the one that jumps out at me as a real opportunity is to create AI-based simulations, behavioral or decision-making simulations, and those can be your medium of evaluation. Where I’ve been recommending scenario-based questions, that’s great, but an AI can create scenario-based questions, a string of them, and analyze that together and dig down deeper than just plain scenario questions on its own. I haven’t seen anybody nail this yet, but I think that’s a real opportunity for us going forward.

Celisa Steele: [00:23:24] In a smaller learning business—I’m thinking just a few people are working in the learning business, pretty modest budget—if that group wanted to make an improvement in how it evaluates its programs this year, what would you recommend?

Dr. Will Thalheimer: [00:23:39] This is an easy one. Most of us are using poorly designed learner surveys. We’re using smile sheets. We’re using Likert scales. We’re using numeric scales. Use a performance-focused learner survey with distinctive answer choices. Those are ones that are more concrete, that learners can get their heads around. They’re more motivated to apply them, and the data you can get can be much more valuable. Most data we get on learner surveys, we get an average between 3.8 and 4.5. We sort of become paralyzed—what’s good, what’s bad—and then we don’t do anything, and that’s no good. But these concrete answer choices are much more valuable because you can see, “Oh, for this question, some people said they’re ready to apply this learning now. Some people said they need more experience. Some people said they’re still confused about the concepts.” Now I have more granular intelligence about how my learning is doing. The number one thing you can do is ask better questions on your learner surveys.

Celisa Steele: [00:24:46] I’d love to hear some of your specific habits, practices, or sources that help you continue to learn and grow—and, in particular, if there’s anything that’s a newer habit, practice, or source that you’re using.

Dr. Will Thalheimer: [00:24:59] I’m a research guy, so I still go to the scientific research. Not everybody’s going to be able to do that. You have to have some skills in that. But that interests me. And so one of the things I’ll say is, “What do you have curiosity about? What can you sustain for yourself? What gets you jazzed?” You’ve got to figure out what works for you. What I do might not work for you. The other thing I do is I’m a big audiobook reader. I listen to books on audio when I go to the gym, when I take a walk, when I’m cleaning the dishes, when I’m cooking dinner, and that’s such a great…. I can’t concentrate on it like I would when I’m reading a book. I can rewind, but I don’t usually. But it gets me thinking about different things.

Dr. Will Thalheimer: [00:25:55] Let me digress a bit. I’m going to oversimplify, but there are two kinds of learning. One is somebody tells you something, it goes into your brain, and now you understand it too. But the other kind of learning is what I call insight learning, where somebody triggers something that makes you think something new. You have a new insight, maybe not the insight that they wanted you to have, but you get these ideas.

Dr. Will Thalheimer: [00:26:17] If you’re constantly feeding your mind with new concepts, new areas of thought, then you’re more likely to have creative ideas, creative insights, and that, to me, is learning. That’s an exciting way to learn when you’re having new insights. Now, I’m four decades into my career here. I’m getting pretty old, so I might not be like your normal listeners, but I’ve begun to focus on other things now as well. I’ve gotten very interested in democracy—and the failures of democracy. I’m reading much less in the learning space and more on reading about the constitution and people’s ideas for how to fix democracy. I’ve become fascinated with that. I still listen to other things as well. Podcasts. Podcasts are a great way to get ideas. I listen to a whole bunch of different podcasts in the social sciences. I’m a big believer in the social sciences because we’re human, and a lot of what we do is based on our humanity. And so those who are studying social sciences and how humans think and learn and get along together and all that really can give us insights about our work and our family life and everything.

Jeff Cobb: [00:27:46] That wraps up the conversation with Dr. Will Thalheimer, founder of Work-Learning Research and author of Performance-Focused Learner Surveys and The CEO’s Guide to Training, eLearning & Work. Stick around for our recap.

Celisa Steele: [00:27:59] On his site, you can find his research and information about his workshops, like the LTEM Boot Camp, and his speaking.

Jeff Cobb: [00:28:14] If you found this episode valuable, please share it to help more people find the show and benefit from the Leading Learning Podcast.

Celisa Steele: [00:28:21] Will made the point that, if everyone is using the same smile sheets, nobody has an advantage. But, if you invest in better evaluation, you create better learning, and that can help you stand out in your market.

Jeff Cobb: [00:28:35] I liked how Will boiled why we do learning evaluation down to three big-bucket reasons: to demonstrate value, to support learners, and to improve your offerings. And he was clear that the last one, improvement, is really the foundation—get that right, and the value becomes easier to demonstrate, and your learners are better supported.

Celisa Steele: [00:28:56] And his LEADS framework gives us a simple test: Before you ask any evaluation question, ask yourself what decision it’s going to help you make. If you don’t have an answer, maybe don’t ask the question.

Jeff Cobb: [00:29:08] Thanks again for listening—and see you next time on the Leading Learning Podcast.

To make sure you catch all future episodes, please subscribe on Apple Podcasts, Spotify, or wherever you listen to podcasts. Subscribing also gives us some data on the impact of the podcast.

We’d be grateful if you’d rate us on Apple Podcasts or wherever you listen. Reviews and ratings help the podcast show up when people search for content on leading a learning business.

Finally, follow us and share the word about Leading Learning. You can find us on LinkedIn.