In our last post, we delved into how people use the term "learning analytics," including an overview of both complexity and category. And this week, we're diving deeper into the four levels of complexity and how they factor into the Watershed Method™.
A Closer Look at Complexity
As you may recall from our previous post, the first dimension in which people talk about learning analytics is complexity—or the sophistication of the analytics.
Furthermore, analytics consist of varying degrees of complexity, which provide richer, deeper insights as higher levels of complexity are attained. And we created the Watershed Method to define these degrees of complexity by breaking them into four levels.
Level 1: Measurement
The most basic form of analytics is measurement—the simple act of tracking things and recording values. Analytics aren’t possible without first gathering this data, which typically occurs through:
- passive data collection, or passively accumulating data as a byproduct of another activity, such as using a system
- active data collection, or intentionally being active in taking measurements to gather data
Level 2: Data Evaluation
Now that you have data, it’s time to start evaluating it, which is the process of finding meaning out of the data you’ve measured. For instance, does your data tell you something something good or bad?
Evaluation usually begins by applying descriptive analytics, which simply means using different techniques to aggregate data to paint a picture of what’s already happening. These analytics help us understand the what questions. In fact, the vast majority of analytics practitioners use simple descriptive analytics.
Common descriptive analytics require relatively basic mathematical computations—such as averages, counts, rankings, ranges, and percent changes—and are often presented in reports, charts, and graphs.
Level 3: Advanced Evaluation
When datasets get large enough,* you can use advanced evaluation techniques—such as exploratory and inferential analytics—to search for relationships within your data and discover powerful insights.
You can then probe these correlations to help form theories about causation and other relationships between variables. And you can start to investigate why things happened, not just what happened.
Keep in mind, advanced evaluation requires trained statisticians to use sophisticated techniques, such as correlation and regression analysis. This level of analysis is often a journey where one bit of information leads to more questions, and where multiple models and visualizations need to be combined to provide insight.
Data mining, artificial intelligence (AI), and machine learning technologies go a step further and can help uncover hidden patterns that humans could never find using manual methods.
Level 4: Predictive & Prescriptive Analytics
The first three levels of the Watershed Method help you understand what’s already happened. The fourth level of the method covers predictive and prescriptive analytics, which help you understand what will happen in the future.
- Predictive analytics attempt to answer the question “given what I know about the past, what is the most likely outcome if Event X happens?”.
- Prescriptive analytics goes one step further. It says, “given that we know the most likely outcome of Event X is going to be Event Y, then when we observe Event X, we should take Action Z to optimize the subsequent result.”
These techniques often require very large datasets* and advanced AI/machine learning algorithms to be effective in making nontrivial predictions.
Applications in Learning Analytics
All four degrees of analytics—and their respective techniques—can be applied to learning in different contexts. In current practice, most learning analytics fall into the measurement and evaluation categories.
In fact, very few corporate learning and development (L&D) departments, use predictive and/or prescriptive analytics. This is mostly due to the lack of available data and of analytical capabilities within the industry. As xAPI continues to expose more data, though, we’re starting to see greater use of advanced analytics.
* The size of a particular dataset will vary depending on what’s being evaluated.
Which way are we looking?
Analytics allow us to look both into the future and into the past.
Measuring leading indicators allows us to evaluate our progress, anticipate our attainment of a future goal, and adjust course accordingly. Educators use "formative assessments" as leading indicators of student success and as a valuable tool for shaping learning experiences.
Measuring lagging indicators allows us to look back and understand whether we've hit our goals or not. Educators use "summative assessments" to evaluate students at the end of a period to assess whether students met the desired learning objectives.
Up Next: Learning Experience Analytics
Next week, we’ll be moving onto learning experience analytics, which help you understand more about particular learning experiences or sets of learning experiences—such as online courses, mobile apps, or in-person classes.
About the author
As an innovative software developer turned entrepreneur, Mike Rustici has been defining the eLearning industry for nearly 20 years. After co-founding Rustici Software in 2002, Mike helped guide the first draft of xAPI and invented the concept of a Learning Record Store (LRS). In 2013, he delivered on the promise of xAPI with the creation of Watershed, the flagship LRS that bridges the gap between training and performance.
Subscribe to our blog