Across the broader business of learning and education, if there’s an area that hasn’t been tapped to it’s full potential – or even to it’s partial potential – it’s educational data mining and learning analytics.
Dr. Ryan Baker, associate professor at the University of Pennsylvania and director of the Penn Center for Learning Analytics, is known for his role in establishing the educational data mining scientific community and also founded the world’s first Masters program in learning analytics. His work focuses on finding student data related to learning, engagement, or emotion that can be used today as well as to predict – and hopefully improve – future student outcomes. He’s published a wide range of academic papers and articles and has also taught the popular MOOC, Big Data and Education multiple times.
In this episode of the Leading Learning podcast, Celisa talks with Ryan about his work in the areas of educational data mining and learning analytics, the importance of understanding student engagement and disengagement, as well as intelligent tutoring systems and their impact on lifelong learning.
Listen to the Show
Read the Show Notes
[00:18] – Thank you to NextThought, the sponsor of the Leading Learning podcast for the third quarter of 2018. NextThought is your partner in learning management system technology, creating engaging experiences, increasing learning potential, participant satisfaction, and member retention. Empowering learning businesses like yours with the perfect combination of art, science, and technology of online learning, Next Thought helps you achieve your education goals. They go above and beyond the standard learning management system by offering comprehensive solutions, including a modern, elegant technology platform, an evidence-based learning design methodology, and professional video production services. Visit http://www.NextThought.com to learn more.
[01:28] – A preview of what will be covered in this podcast where Celisa interviews educational data mining expert, Dr. Ryan Baker, associate professor at the University of Pennsylvania and director of the Penn Center for Learning Analytics.
[02:55] – Introduction to Ryan and some additional information about his work including that his focus is on trying to figure out what’s going on now in a student’s learning, engagement, or emotion that’s predictive of where they’re going to end up in the future. So figuring out how to help them today so they can have a better outcome tomorrow.
[04:14] – Your work focuses on educational data mining (EDM) and learning analytics—which are related but different. Would you briefly explain the key differences and similarities? Ryan points out that the similarities between the two are more important than the differences. He says they came out of two different crowds of people and the educational data mining community (which started a little earlier) came out of people interested in themes in K-12 and scientific discovery. On the other hand, the learning analytics crowd came out more from higher ed with more of the idea of supporting the practice by informing them. There are a few differences including that EDM is a little more focused on automated intervention and adaptive systems. Learning analytics is a little more focused on informing instructors/coaches on how to better support learners. However, broadly they’re both concerned with key themes of what we can take from the increasingly large data available about learners and learning and using it to make a difference in some fashion.
[05:36] – What do you see as the most exciting potential for educational data mining and learning analytics, and, on the flip side, where do you have reservations or concerns about how they’re being used or might be used? Ryan thinks there’s a lot of areas of potential which is what’s exciting to watch in the space. Over the past five years, adaptive learning has become much more widespread, instructors and various folks at universities have reports about which students are at risk and why, K-12 guidance counselors/teachers/administrators have reports about which students are at risk—and we’re really starting to see these things scale up. These days, most K-12 students are in schools using predictive analytics to try to figure out which kids are at risk and why. He says the success comes with a really big challenge right away because you can use the data really effectively or use it really poorly. The quality of the models/algorithms we’re seeing is quite variant. There’s a ton of promise in this data but we have to really question whether the algorithms are good and what our basis is for believing that. Until we can get to a point where everyone out there is at that level of quality we won’t be able to know if we’re reaching our potential.
[08:08] – I hear a lot about learner engagement—disengagement not so much, and yet it’s the obvious the companion to engagement. How do you define or describe disengagement, and what have you learned about it in your research and work? Ryan explains there are so many ways a learner can become disengaged and they have very different implications. Part of what he thinks we need to do is not think of disengagement entirely as a problem – or engagement as entirely good – but think of how a student is engaging. He shares an example to highlight that breaks aren’t a bad thing and can be good for a student’s overall engagement. In contrast, gaming the system is a more often ignored disengaged behavior. Yet research shows that students who game the system in middle school math, for example, are less likely to go to college and less likely to major in a STEM career.
[10:04] – How do you begin to tease out those different flavors of disengagement (i.e. someone trying to game the system versus people using off task behavior as a way to reset and learn more)? As far as detecting things like gaming a system, Ryan says they’re able to do it from system logs. They can detect off-task behavior from system logs as well but telling what a student is doing off task, beyond just the duration, you need to bring a human being into the loop. Eventually we may be able to use sensors/cameras for that but there’s a lot of concern at a national level around this. So if a student is going off task a lot, Ryan recommends getting the guidance counselor and teacher involved to think about what it means—and communicate to them that a little bit of off-task behavior here and there actually isn’t that big of a deal. Ryan adds there’s an increasing awareness in the learning analytics and EDM community that it’s not all about the fanciest algorithms but about taking what human beings are good at and taking what computers are good at, and trying to leverage both.
[12:20] – Intelligent tutoring systems is another area of focus for you. Can you tell us a little bit about how you define or what fits into intelligent tutoring systems and then what you see as the application to lifelong learning, outside of academic setting? Ryan shares that intelligent tutoring systems have been around for a long time but what they’ve increasingly realized is the strengths of an expert human tutor and the strengths of a computer aren’t the same. As an intelligent tutoring system we need to continually figure out what a student is struggling with, where they need support, and what kind of support they need. All good intelligent tutoring systems have a model of where a student is that’s multidimensional and captures more than just what they’re able to get correct right now and some pedagogical practice that’s built around that—and automatically adaptive. As far as it goes for lifelong learning, there’s been a lot more work on this in higher ed and K-12 than there has been in the lifelong learning space. For example, Ryan says with the generous support of the US Army, they are actually trying to build intelligent tutoring support into his MOOC. He also talks about the value in intelligent tutoring systems being able to connect different learning opportunities (not tied to one specific course) that can be informed about what the learner is doing across the spectrum of different learning activities.
[15:10] – You’ve done work applying intelligent tutoring systems to MOOCs—is that the key to personalized learning at scale? Ryan explains there’s not really a distinction to the terms “personalized learning”, “adaptive learning”, and “intelligent tutoring systems”, and they’re actually all the same thing. Ultimately, it’s an idea of trying to take a system that can figure out what the differences between students are that figure out their needs and provide something different to them. He says that technology is a lost opportunity that we’re not seeing in MOOCs yet.
[16:32] – For our listeners, leaders and aspiring leaders working in lifelong learning, and folks who may not have access to tons of data and resources, what do you see as the main practical take-aways around learning analytics, educational data mining, intelligent tutoring systems, your work in general? What should they be considering or doing? Ryan recommends considering how you can build in the best technology that’s out there. If you can bring in a good system (or a predictive analytics system), do so but try to make sure what you’re getting is good because there’s a lot of snake oil out there in the midst of some very good technology as well.
[17:48] – Do you have any tips or suggestions for how people begin to tease out those good quality systems and technology that you’re mentioning versus the “snake oil” that you mentioned? Ryan says one of the key aspects is systems that are good typically have some trail record of publication or solid peer-reviewed evidence behind their effectiveness. A lot of the people that are developing good technology are putting it out to the world to be checked or commented on. So the presence of peer reviewed publications and respected venues are a key indicator. He also thinks among the systems that don’t do that, there’s different kinds of argumentation—if the kind of argumentation you’re seeing is based on technical arguments that are detailed, that’s probably a much better sign than vague arguments or grandiose claims.
[19:26] – What’s on the horizon for learning? Are there any big developments or changes you think we’ll see in learning in the coming years? Or that you hope we’ll see? Aside from the growth towards greater personalization, Ryan says the capacity to produce learning experiences that are richer and more experiential in various ways is a really powerful aspect of the upcoming generation of learning technologies. So rather than just watching a video, reading a text, and then taking a quiz, people are engaging in complex problem solving, using rich simulations, and engaging in virtual or augmented reality. These kinds of developments will enable people to learn things that they can connect much clearer to the context they care about.
[21:17] – What is one of the most powerful learning experiences you’ve been involved in, as an adult, since finishing your formal education? Ryan shares about his experience trying to learn Korean (his current learning project) where he realized none of the technology tools out there had the flexibility he wanted. He ended up not using any adaptive learning tools but this made him realize how incomplete solutions are and that the real future of learning is going to be in creating learning interactions that provide a comprehensive way to learn things.
[23:19]: How to learn more about Ryan and his work:
- University of Pennsylvania Website: here you can view a range of Ryan’s publications.
- Twitter: @BakerEDMLab– a low traffic Twitter feed where they post all of their latest scientific results. On that page, there are a couple of review articles that give a big picture. Specifically, he recommends this audience checking out the article, Using Learning Analytics in Personalized Learning.
[24:33] – Wrap Up
If you are getting value from the Leading Learning podcast, be sure to subscribe by RSS or on iTunes. We’d also appreciate if you give us a rating on iTunes by going to http://www.leadinglearning.com/itunes.
And we’d be grateful if you would take a minute to visit our sponsor for this quarter, NextThought. We put a lot of time and energy in to the Leading Learning Podcast, and one of the key reasons we’re able to do that is because of the support of sponsors like NextThought so please visit NextThought.com.
Also, consider telling others about the podcast. Go to http://www.leadinglearning.com/share to share information about the podcast via Twitter, or send out a message on another channel of your choosing with a link to http://www.leadinglearning.com/podcast.
[26:29] – Sign off