Learning Analytics and Using Data Responsibly

Last week, I stumbled on two media sources surrounding the ethics and responsibility of data collection that made me think a bit more in-depth about learning analytics and the data I am collecting from the online portion of my first-year seminar course (more about that here and here). The first was an interview from a podcast that I’m subscribed to (and which I am also a little behind on!), “The Internet of Things Podcast” with Stacey Higginbotham and Kevin C. Tofel. The episode I listened to was 63, which included a really great interview with Ernesto Ramirez about the potential for wearables to become part of our health records in an impactful way. Ramirez and Higginbotham discussed the ethics around collecting, sharing and acting on data from fitness trackers (think FitBit, Garmin, Misfit, Jawbone, etc.).

One poignant piece of the conversation that got me thinking about my own project was the discussion around whether it was fair to give employers access to employee health data because they could potentially use it to make business decisions that ultimately end up hurting the employee. For example, someone who logs the regular 8 hours of sleep per night might be favored for a big assignment at work as opposed to someone who doesn’t get that much sleep, with the thought being that the person who gets more sleep would perform better. Obviously, anyone in the working world knows that everyone is different and sleep needs vary.

Another really interesting point was briefly mentioned: how accurate are these fitness tracking products, and who is liable should their data ever become a major component of our health records? Could a doctor who “misses” a sign from tracker data in someone’s health record ever be held accountable if something bad happened to their patient? Would that be the manufacturer’s fault, or the doctor’s?

These were all really interesting things to think about in the context of learning analytics. Of course, learning data is a fledgling field at best, and it’s no where near as life-threatening as health data could be. However, early warning systems that would help us identify faltering students before they’ve dug themselves in too deep offer the same question: who’s responsibility is it to “catch” a falling student? Faculty? Advisors? The students themselves? I have no idea, but it’s a conversation worth having if your institution is looking into any system that offers this level of feedback!

Conveniently enough, I was talking about the IoT podcast interview with some ed tech friends at work, and one of them shared with me this blog series by a student at a Canadian university. Out of curiosity about the impact of data on his final grade in an online course, he looked into the school’s policy about course data retention, filed a Freedom of Information request, and eventually got his data out of the school’s CMS, Blackboard Connect.

As the student, Bryan, began the process of filing the formal request and working with the university to get his data, he started posting about the hurdles and barriers that had to be overcome in order to even access the data. Along the way, he was able to meet with administrators that operate the school’s instance of Connect and learn a little about the way the system collects data and what instructors have access to when they are teaching using the product. His blogs were a huge insight into the way some students feel about their actions being tracked online, and a reminder that the reaction is not always positive.

These days, I feel like it should be expected that everything we do online is being tracked in some form or fashion. We see it in our customized ads on the social sites we use, and we see it in our email inboxes when we swipe store rewards cards or use apps to make purchases. So why is it so jarring to students when they do learn that everything they do in the online learning space is tracked too? One could make the argument that in some cases, customized ads and product suggestions are helpful, or at least aiming to be. I would surmise from reading Bryan’s blogs that the intention of CMS data tracking came off as somewhat sinister at first, with it being tied to performance in a course. The same logic applies to the fitness tracking data example — when we can use the data to be informed about our health patterns, just for our own personal benefit, everyone who purchases these devices is obviously fine with it (and some of us are quite dedicated to improving based off the information we get from these gadgets). But the idea of an employer basing decisions on it turns things sour really quickly, doesn’t it?

So would students feel differently if they could get relevant suggestions, like study tips, based on their data in each class? Would it be helpful for them to see where their weaknesses and strengths lie, so they could know where to focus their energy? I would love to see learning data get to the point where each student could download a historic record of their own patterns of behavior and be able to predict where they might stumble with certain courses based on the type of content covered in that class, or know ahead of time that they tend to fall behind at certain points in a semester. Universities are about educating students and empowering them, and wouldn’t providing them with their learning data fulfill that mission?

Aside from the benefit to students, I would love to see faculty using data in a new way. I have a strong stance on participation grades and why they aren’t the best assessment strategy (which is a conversation for another post), so rather than studying students’ behavior as a reflection of their performance, why not instead try to find out what their online access says about the instructor’s performance instead? For example, if a course is really media heavy, with a ton of videos that contain all the direct instruction and which took a lot of time and production to create, it would be possible to see whether those videos were performing the way they were intended to by looking at how frequently students watch them. If your students aren’t watching your videos but they’re still succeeding in your course, it’s time to stop investing so much time into those videos and figure out what the students are doing that is working for them instead.

As a designer, there are limitless possibilities for real time data collection — I can know right away if the navigation and progression of a course are working for the class, or if adjustments need to be made that will help students be more successful in the next week or in the next unit, rather than waiting until the next time the course runs to try something new. I could see the data from an entire course and be able to use that information to say “Wow, the content from Unit 3 and Unit 5 got a lot of activity this semester, maybe we need to look at massaging that content to make it easier to digest next time!” or that students whipped through units 1 and 2, and maybe we can combine that information into one unit and expand the amount of time spent on the more difficult topics.

Those are the things that are possible with good learning analytics, and that’s what professionals in the field should be focused on making happen — not wasting time on these ridiculous participation grading formulas based on number of logins over a period of time. As long as we in the industry are upfront and honest with our intentions for collecting data about student interactions in the online space, and that we use the data for the purposes of improving educational outcomes for all stakeholders (students, faculty, designers, administrators, etc.), I don’t see how anyone could argue that collecting all this data is a bad thing.

Feel free to share your thoughts on this in the comments, or reach out to me on Twitter @katrinamwehr!