Steven Khraiss
Many learning businesses evaluate their programs with data that only tells them what already happened. By the time a lagging indicator like a completion rate reveals that a learner is struggling, the window to help has often closed. The good news is that shifting to real-time and predictive analytics is readily achievable for most learning businesses.
In this episode of the Leading Learning Podcast, co-host Celisa Steele talks with Steven Khraiss, who heads data strategy and analytics at Holmes Corporation, about the ideas behind the e-book The Association Leader’s Guide: How to Turn Learning Data into Personalized Member Experiences. Steven explains what an engagement score is and what it enables, why there’s no mention of AI in the e-book even though it’s powering a lot under the hood, and why he names culture, not technology, as the harder challenge in becoming data-driven. He also offers a practical first step for any learning business leader who wants to start but doesn’t know how: Start with one question you wish you could answer.
To tune in, listen below. To make sure you catch all future episodes, be sure to subscribe on Apple Podcasts, Spotify, or wherever you listen to podcasts.
Listen to the Show
Access the Transcript
Download a PDF transcript of this episode’s audio.
Read the Show Notes
Celisa Steele: [00:00:03] If you want to grow the reach, revenue, and impact of your learning business, you’re in the right place. I’m Celisa Steele.
Jeff Cobb: [00:00:10] I’m Jeff Cobb, and this is the Leading Learning Podcast.
Jeff Cobb: [00:00:17] Too many learning businesses are trying to evaluate their success by looking in the rearview mirror. By the time the data shows a learner is struggling, it’s often too late to help.
Celisa Steele: [00:00:26] In this episode, number 485, I talk with Steven Khraiss, who heads data strategy and analytics at Holmes Corporation. Our conversation is structured around a Holmes e-book, The Association Leader’s Guide: How to Turn Learning Data into Personalized Member Experiences.
Jeff Cobb: [00:00:43] Steven makes the case that the shift from lagging indicators like enrollment counts and completion rates to real-time and predictive analytics is achievable for most associations. And the cost of waiting is rising faster than many leaders realize.
Celisa Steele: [00:00:57] Steven and I get into how to create an engagement score and what it enables, why Holmes deliberately left any mention of AI out of the e-book even though AI is powering so much under the hood, and why Steven names culture, not technology, as the harder challenge in becoming data-driven.
Jeff Cobb: [00:01:15] And Steven offers a practical first step for any learning business leader who sees the value but doesn’t know where to begin: Start with a question you wish you could answer.
About Holmes Corporation and Steven’s Role
Celisa Steele: [00:01:31] Steven, for listeners who may not be familiar with Holmes Corporation (HC), give us a quick sketch of what you all do, who you serve, and how you serve them.
Steven Khraiss: [00:01:42] Yes, absolutely. At Holmes Corporation, we partner with professional associations to design, build, and distribute knowledge and skills assessments—our flagship certification preparation. But we’ve also branched out into microcredentials as well as other learning programs globally. The goal is driving member engagement and revenue at scale. It’s turning those learning programs that actually move careers; that’s the gap we fill. We partner with some of the largest in the world: SHRM, ASCM, ICF, APIC, AFCPE, PayrollOrg. But the differentiator there is we span many different verticals—HR, supply chain, coaching, healthcare, finance—and everything from market analysis, content, marketing, sales, ongoing customer care. We’re not just a platform vendor. We really try to help associations meet their end goals. My role in this data and analytics layer sits underneath all that, and that’s the part that lets our partners actually see what’s working and what’s not. We keep evolving as our association partners’ needs do.
Three Trends Driving the Growing Importance of Learning Data
Celisa Steele: [00:03:04] Speaking of data, the impetus behind this conversation is an e-book that Holmes released called The Association Leader’s Guide: How to Turn Learning Data into Personalized Member Experiences. When I read that publication, it mentions three converging trends that are driving the importance of having good learning data now: the rising accountability expectations from stakeholders, the reskilling imperative that’s creating demand for learning programs, and then external and competitive pressure to keep pace. If you had to pick one of those, what do you see as the biggest driver of urgency right now for learning businesses? Or maybe it’s something other than those three at this point? I would love to hear what you think is driving urgency now.
Steven Khraiss: [00:03:55] Absolutely. All three are very real. But, if I have to pick the one that’s creating the most urgency—not only this year but over the past five years—it’s accountability. Association boards or enterprise partners, just association partners themselves, are asking a sharper question now of, “What did we actually get for this investment?” As budgets become tighter and every learning dollar, and in this role, having to balance a budget, it’s harder than it was years ago. But simply saying “Enrollment was up” is not the acceptable answer anymore. And the reskilling wave is real. We quote some stats in the e-book, but the World Economic Forum says 59 or 60 percent of workforce needs reskilling by 2030, and 95 percent of organizations today have ongoing and upskilling efforts underway, per Workday’s 2025 data.
Steven Khraiss: [00:04:59] In my mind, the demand is not theoretical anymore. And regarding competition, we frame it in the e-book well, but the gap between the real-time insights and a static quarterly PDF, or weekly, is the difference from knowing how the training is working and hoping it does. But they’re all pushing in the same direction. I’m living this firsthand when rebuilding our data foundation and infrastructure and having to connect different systems into a unified data lake house, so we could answer, “What did we actually get out of this investment?” and with evidence, not just “Well, we got some cool things.” When I say accountability is the driver, it’s what we built for.
The Risk of Relying on Lagging Indicators
Celisa Steele: [00:05:51] You were talking there about some of the things that, historically, organizations have relied on, things like enrollment counts and completion dates. But the e-book makes the point that those are lagging indicators, and so they give you limited visibility into is this program working and making a difference? What are the risks when learning businesses rely on those kinds of data, those lagging indicators, to tell them, “Is this working?”
Steven Khraiss: [00:06:22] Excellent question. The core problem is timing. Like I mentioned, the numbers tell you what already happened if you’re not working with real time, and so, by the time something looks bad, or you see learners’ progression fall off, you’ve usually already lost them at that point. While “Enrollment was up 10 percent” sounds great in a board deck or displays well on a report, it doesn’t really tell anything of whether anyone’s learning anything within a core set of products. In the e-book, we break it down into four costs. Delayed intervention—you spot a struggling learner that’s maybe too late to help. Limited accountability—it’s hard to prove ROI to leadership with just those surface-level metrics. Three, the resource drain—having to pull (I’ve lived this again) the manual reports and trying to chase learners one to one doesn’t scale. And four is those missed broader patterns of you can see what’s working, what’s not with learners, and what are they actually doing within products. Every one of those four problems is what that real-time or near real-time visibility and predictive scoring are essentially built to solve. And so relying on sports references, it’s like coaching a game off of the final score. It’s accurate, but it may be too late to change anything versus having to go back and review it.
The Value of Moving to Real-Time and Predictive Analytics
Celisa Steele: [00:08:04] You began to talk there a little bit about those real-time but then, even beyond that, predictive analytics. What does it look like when a learning business shifts from that reactive lens to a predictive lens? And what does that mean? I’m thinking internally for staff at the learning business and also for those learners that learning businesses serve.
Steven Khraiss: [00:08:26] Reactive is reading the autopsy, and predictive is reading the vital signs. For it to work, the model needs history, some of those validated signals (the key performance indicators), and then the outputs that translate into the next step and not just a scary number. For staff, the shift is to viewing reporting in a proactive approach and doesn’t just replace it. For learners, it helps show, for example, if they’re unengaged. It’s going deeper than a “Hey, we miss you” message versus where they are in their journey. And we’ve seen this pay off in our own analysis and data. From a member and learner perspective, we analyze thousands and thousands of learner records across various dimensions, membership status. Typically, our products will have an initial pretest, so we measure engagement there. And then, if it’s a cert prep product, a practice exam—we measure engagement with those. And then overall engagement with the product. We’ve seen engagement in the practice exam behavior as one of the single biggest levers. We can design nurture programs that say, “Hey, we noticed that, Celisa, you plan to take your test in two weeks. It’s very vital that, if you haven’t yet, please take your practice exam. It’ll give you that peace of mind and framework that really makes a difference.”
Celisa Steele: [00:10:07] One of the things that the publication talks about is this idea of an engagement score. For those individual learners, you have this idea of how engaged are they? That’s, I think, based primarily on activity in the platform, logging in, and have they completed certain modules, those sorts of things. Talk a little bit more about the engagement score—how it works but, even more importantly, what having that engagement score then enables for an organization.
Steven Khraiss: [00:10:34] At the simplest level, the engagement score can answer one question extremely fast: In your ecosystem of learners, who’s thriving, and who’s quietly slipping away? It’s built off of those behavioral signals, like you mentioned: platform activity, login frequency, how long per session are they engaged? Is it 10, 20 minutes versus two and a half hours? How are they moving through the various modules and, with that, their assessment activity—whether it’s taking quizzes, or if we have flashcards built into the product, or downloading various items? When you consolidate all that and put it together, that engagement score is reflecting the likelihood of finishing the program successfully, whether that’s ultimately taking a certification exam or earning a microcredential. What it really unlocks at scale is, for the exam, without having to dig into individual learner trends, it helps you see the overall macro picture. But I always have one honest caveat there: The score is only as good as the behaviors you feed it. Keep it tied to real learning, not vanity clicks. The mental model I use is to think of the credit score for engagement, one number, many different behaviors, and it moves and shifts over time.
Celisa Steele: [00:12:02] Talk about what the engagement score is. Is it a scale? Is it a number between 0 and 10? Are you categorizing learners into specific tiers? Talk a little bit about that.
Steven Khraiss: [00:12:16] Absolutely. We’ll look at the wider dataset based on trends. Whether engagement with nurture programs, whether it’s an e-mail campaign, or an SMS-triggered campaign, we’ll assign various scores. And they’re mostly weighted. If we know, for instance, a practice exam is a high driver, then we might give a learner a score of 20 points. You could set your scale to whatever is relevant to the business, but then you create these tiers of low engagement, middle, high, super high, or super user. Time and time again, we’ll run these analyses, and we always say around the office, “If you use the learning system or our products as they should be intended, you will have a higher likelihood of success.”
Celisa Steele: [00:13:11] In essence, that engagement score then gives the provider of that content a shorthand way to see who is most engaged—perhaps offer encouragement to those who aren’t as engaged, maybe offer kudos to those who are very engaged, and so it’s about providing the appropriate feedback and incentives along the way to keep the learners going.
Steven Khraiss: [00:13:35] Yes, you just hit something right on the head there. We’ll have tap-on-the-shoulder nurture campaigns where a learner may be interactive and engaged early on, say, for a certain time period, and then they fall off. Life happens. Or you get down this runway of “Yes, I’m going to really engage here.” And so we have triggered campaigns that say, “Hey, Celisa, you’ve been absent for a couple of weeks. Let’s get back and focus and do x, y, and z.” Likewise, if a learner is really engaged, we have campaigns that praise them along the way. It’s that full end-to-end on a journey. It’s almost like a coach that’s cheering you on throughout a game or throughout your sports journey.
Celisa Steele: [00:14:25] I see you’re back to sports again there, Steven.
Steven Khraiss: [00:14:26] Yes.
Why AI Isn’t Mentioned Explicitly
Celisa Steele: [00:14:28] In the publication, there’s no mention of AI, which is a bit surprising, given that this is a piece focused on learning data, on personalization. I want to give you a chance now, Steven, to go on record and talk about the role of AI in making effective use of learning data.
Steven Khraiss: [00:14:49] I’m really glad you’re asking this one. Honestly, it was a strategic decision. We left off the word AI throughout the piece on purpose because the fact is it’s already in there and not just as the buzzword of AI. Because, in my mind, most association leaders don’t really want AI for its own sake. They want the outcomes that it delivers. When you look under the hood of the e-book, it’s all there. When we talk about predictive risk scoring, the engagement score, behavioral automation that fires off the personalized nudges based on how the learner is interacting, that is essentially the machine learning doing the useful work quietly in the background. And through those automation processes and workflows, we’ve seen real results. We’re cutting our partners’ workload by, in one instance, over 45 percent, and we’ve seen more than three times the industry average in terms of open rates and click rates with our e-mail campaigns.
Steven Khraiss: [00:15:53] We chose to lead with the result instead of the buzzword. The flip side is the guardrail. AI is obviously an accelerant but not always the foundation. If the data is fragmented and ungoverned, then AI will scale that mess faster. The common analogy is garbage in, garbage out, but AI at machine warp speed. Even though I’m a huge tech guy, I feel like the technology should disappear behind the result. In my mind, if a member feels seen, it doesn’t matter if it’s AI or a person—that’s generally the whole premise. AI can be the accelerant but not necessarily the foundation. And the associations, at least in our experience, that win won’t have the flashiest AI, but they have the clean data to trust.
How to Benchmark Well
Celisa Steele: [00:16:51] That’s a really important point about the data that goes into any model, especially an AI model. It needs to be in good shape if you’re going to expect it to be able to help you generate meaningful results. One of the other things that comes up in the e-book is benchmarking as a way to use some of the learning data. That’s interesting because benchmarking can be really powerful and can be useful. But it also seems like sometimes benchmarking can create a bit of uneven comparison, like an apples-to-oranges problem. What does it take to benchmark meaningfully, and what makes those benchmarking comparisons valuable and useful rather than misleading—being compared to not a similar organization or a similar situation or a similar learning product?
Steven Khraiss: [00:17:46] Benchmarking is one of those things. It’s incredibly powerful when it’s done right. And, on the flip, it can be quite dangerous when it’s done wrong because, like you alluded to, a bad comparison can give you false confidence in a dataset that you’re looking at. Performance data is definitely valuable, but performance data with context is even more powerful. I would say make it real. First is the breadth of context—the comparison has to be grounded in data that spans, whether it’s learner groups or industries. If it’s too narrow, it can mislead. Second, if you’re comparing similar courses or benchmarking for us different organizations or cohorts, that makes the peer comparison done right. The key with all of that is the consistent definitions part. Completion has to mean the same thing on both ends of what you’re comparing, and historical trends can also show you the real improvement over time. Often, even in our leadership meetings, it’s like, “Okay, well, x percent is really great, but what does that really mean in context?” The differentiator for us is understanding how adult learners and customers engage and not just the technology that tracks them. Benchmarking is truly useful when you’re comparing the like to like. Otherwise, you can generate very confident but also wrong conclusions.
Celisa Steele: [00:19:28] That’s an interesting thing that Holmes Corporation—because you work with a variety of different organizations—does have that ability to draw on data from different organizations and do some of that benchmarking of program to program, organization to organization. Do you also find at times that it’s useful to benchmark within a singular association, to look at their own program A versus program B versus looking at association A and association B?
Steven Khraiss: [00:19:56] Yes, that’s exactly it. The nature of our association partners in different verticals allows that high-level view. But, to exactly your question, drilling down within a particular industry, within a particular partner across products, it is interesting. You’ll see different trends where, even if the products are designed similarly or with similar functions based on the content, they interact completely different. One of the biggest values for us is being able to do those comparisons and looking at equal time frames and those other environmental factors as well.
The Connection Between Membership and Learning
Celisa Steele: [00:20:46] One of the other things that I found very interesting in the e-book is this finding around the connection between membership status and learning. Could you talk a little bit about this value of “membership-integrated learning”? I think that’s how the e-book put it.
Steven Khraiss: [00:21:00] Yes. In our experience, here’s something associations have always believed but couldn’t always prove: When we look across partner programs in our own analysis, time and time again, members outperform non-members on success rates even before you factor in product engagement. We know association partners that we work with have so much knowledge and industry authority that it becomes this virtuous cycle where membership can unlock the learning, the learning drives engagement, and then engagement ultimately drives renewal and credential revenue. When you finally get that one and unified view of the member, the whole journey—instead of having learning sit in one silo, membership in another—that single consolidated view is where the personalization starts. It means you can recommend the next relevant product or credential and not truly blast the whole catalog. Learning is one of the stickiest reasons a member renews. You weave it in. And every course can make someone engaged, more credentialed, and ultimately more likely to stay.
The Tech and the Culture Needed to Shift to a Data-Driven Approach to Learning
Celisa Steele: [00:22:19] Like you said, it’s often talked about—that relationship between membership and learning and how they are this nice, virtuous cycle if you get them going. As you said, to have some of that data, to be able to look at that and then prove that that’s more than a hunch or the way that it might work, that it actually is the way that it’s working, that those people who are members do seem to engage more or do better in the learning. One of the other things that the e-book talks about is what it takes for an organization to be ready to shift to this data-driven approach to learning, and there are at least two parts to that. There’s at least the technology side of it, and then there’s at least a cultural part to it as well. In your experience, is it the technology issues or the cultural issues that make it harder to shift to a data-driven approach to learning?
Steven Khraiss: [00:23:15] I love this question. As I mentioned, I’m a tech guy, so people that know me would expect me to say “technology,” but I feel that it’s really not. It’s culture every time—the e-book talks explicitly about this. And technology alone isn’t enough. We outlined some of the readiness pillars near and dear to my heart. Data literacy—can your stakeholders read what’s on these dashboards that you’re creating and understand it? Process adaptation—do the workflows change around insights? I talked earlier about actionable insights from the data. Organization-wide or cultural commitment—do you act on what you see? The technology is absolutely necessary, but those three are what separate different organizations that truly win.
Steven Khraiss: [00:24:13] When you have a dashboard that contradicts a senior leader’s gut intuition, the room might just default to that, but I would say that’s the cultural gap, not a technology gap. And so my favorite line is “It takes leadership to model that behavior.” Once teams really trust the data, then there’s this shared willingness to let the numbers challenge all the everyday assumptions. And then it creates this culture that makes it safe to say, “Oh, yes, I was wrong.” You can buy a platform, but you can’t buy willingness to trust in the numbers when it tells you something that you didn’t necessarily want to hear.
How to Get Started on Making More Data-Informed Decisions
Celisa Steele: [00:24:56] For a learning business leader who is going, “Okay, I get it. I understand the value of becoming more data-driven, data-informed,” but maybe she’s not sure where to begin, what would you recommend as a first step to get a little bit closer to being more data-informed/data-driven about decisions?
Steven Khraiss: [00:25:18] I get asked that question a lot in my journey. The most common mistake is thinking you need, out of the gate, a giant platform and this brilliantly designed master plan. You really don’t, and I’ll get to why I think this way. Don’t start with the platform, but start with a question you wish you could answer. The momentum, in my experience, ultimately builds that master plan. You have the framework and a roadmap for sure, so pick something that is concrete and that you wish you could answer. Which of our learners or which segments are most likely not to finish, and why? And then you chase the data to answer that. You don’t have to have everything unified on day one to start out. The e-book provides a little checklist of starter questions. Can you get to real-time data without manual report pulls, or can you predict risk with enough ramp and lead time? Can you automate support so you’re freeing up organizational-wide time? Do you have that peer benchmarking for context? Any one of those is a fine front door. The reason I push more for that one question is momentum, and each small win builds that larger belief and, quite frankly, the budget for a larger investment in the future.
Celisa Steele: [00:26:51] That answer reminds me of AI not being apparent in the e-book. It’s there in the background. Having a question to start with echoes that it’s about the purpose—what are you trying to do—rather than focusing on the tools or the specific methodologies.
Steven Khraiss: [00:27:07] Yes, that’s a great point. Just like AI, it starts with a prompt. You might know where you’re trying to go, but you build momentum with, “Oh, I didn’t think of it in this way.” So it leads it with that excitement and that full end-to-end journey.
Steven’s Approach to His Own Learning
Celisa Steele: [00:27:25] Because this is the Leading Learning Podcast, we always like to ask guests who come on about their own approach to learning. Steven, what are some of your specific habits, practices, or sources as you continue to grow and learn professionally and personally?
Steven Khraiss: [00:27:42] One of my favorite learning experiences, most recently, has been coding with my son. He’s big into Roblox. I know it sounds silly, but wrangling that with him honestly sharpens how I think about logic and systems in ways that my day job doesn’t, even though I am a tech geek. But that’s my whole approach. I learn best hands-on, building things. And projects honestly teach me a lot more than maybe a certain course would, just with that hands-on approach. I deliberately cross-pollinate—you’ve heard that I’m a sports guy—so I pull a ton. I used to coach my daughter’s basketball team, so I like to think of data in that context, as with teams. I try to pull from a lot of different perspectives before locking into my first instinct. And I always try to maintain a thirst for additional knowledge. I listen to podcasts, attend Webinars, conferences—probably too many. I’m on a lot of newsletter lists, and I always try to stay up to date on the latest trends and even test drive new products that interest me. If I had to sum it up, my best learning habit is staying a little uncomfortable because, if I’m in a room, and I understand everything, then I’d be in the wrong room.
Wrap-Up and Recap
Celisa Steele: [00:29:18] That wraps up my conversation with Steven Khraiss of Holmes Corporation. Stick around for our recap.
Jeff Cobb: [00:29:24] Check out the e-book from Holmes Corporation and Steven on LinkedIn. And please help us spread the word about the Leading Learning Podcast.
Celisa Steele: [00:29:39] Steven believes learning businesses need to stop evaluating programs based on data that only tells them what already happened. By the time a lagging indicator like a completion rate reveals a problem, the window to help learners has often closed.
Jeff Cobb: [00:29:55] Holmes made a deliberate choice to leave mention of AI out of the e-book entirely, not because AI isn’t relevant but because, as Steven puts it, AI can be the accelerant, but it can’t be the foundation.
Celisa Steele: [00:30:08] His practical first step for any learning business leader who wants to become more data-informed but doesn’t know how to start is to begin with one question you wish you could answer, and build from there. I love that recommendation—because he’s saying, essentially, start with curiosity.
Jeff Cobb: [00:30:25] Thanks for listening—and see you next time on the Leading Learning Podcast.
To make sure you catch all future episodes, please subscribe on Apple Podcasts, Spotify, or wherever you listen to podcasts. Subscribing also gives us some data on the impact of the podcast.
We’d be grateful if you’d rate us on Apple Podcasts or wherever you listen. Reviews and ratings help the podcast show up when people search for content on leading a learning business.
Finally, follow us and share the word about Leading Learning. You can find us on LinkedIn.
Up Close with an Award-Winning Learning Business
Leave a Reply