We've outlined seven practical steps when evaluating learning programs. Here, we'll discuss the second step in the learning evaluation process—identifying the success metrics most appropriate to the organization.
As we discussed in our last blog post, the first step for evaluating your learning programs is to align program goals with strategic priorities. With the program goals aligned, now it's time to divide them into individual evaluation metrics to show the program's success and monitor progress toward the program's goal. To some extent, this step happens in tandem with Steps 3 and 4 (Discovery and Design), as the information gathered and decisions made in these steps affect one another.
How Do I Select Performance Metrics for Learning Evaluation?
When you define metrics, we recommend a collection of metrics that cover as many of the five areas below. Each program goal should have one key performance indicator (KPI) that can be monitored throughout the program. These KPIs will be the most important metrics of your evaluation.
The following list provides examples of evaluation metrics at each level:
1. Usage and quality of content
- Number of times content is accessed*
- How much and which parts of video content are skipped
- Attendance at events
- Time spent
- Difficulty and quality of particular assessment questions
- Survey responses
2. Delivery and discoverability of learning experiences
- Number of times the experience is experienced*
- Clicks or time taken to access the learning experience
- Survey responses
3. Improvements in knowledge and competency
- Assessment/simulation scores comparing progress before and after the learning experience
4. Improvements in job performance
- Peer/manager/mentor observations
- Customer feedback
- Data from business systems used to do the work
- Survey responses
5. Overall achievement of the program goals
- Number or value of sales made per month
- Average time taken to produce each widget
- Average customer satisfaction rating
- Employee retention %
*Number of hits applies to both levels as the reasons for high or low numbers could relate to either quality or discoverability. This metric needs to be used in conjunction with other data or research.
Expressing L&D Metrics
The specific metrics and how they're expressed will depend on both the program and the organization. This is particularly significant for the metrics relating to the overall program goal. For example, if an organization is interested in short-term benefits and has confidence in the value of learning, the metrics might stand on their own. In an organization that's more interested in long-term benefits and has low confidence in the value of learning, the metrics should be expressed in relation to industry benchmarks.
Whatever metrics you choose, plan to measure them over a period of time. It’s important to evaluate whether or not learners remember their learning and persist in their changed behaviors.
If you want to know if your learning program is successful, you must first determine threshold values for each metric to represent results that are expected, below expectations, or above expectations. Keep in mind, these thresholds may change. For example, as your organization improves. Remember to link these thresholds to industry benchmarks or levels of return on investment/expectation as appropriate for your organization.
In addition to those metrics, define a set of quality criteria for your materials and resources, which can be applied before learners interact with them. Criteria should reflect both production quality (e.g., Are images used appropriately?) and pedagogy (e.g., Does the solution include reminders to aid remembering?).
Once you're satisfied with your chosen metrics (and how they're expressed), review them with relevant stakeholders to make sure the metrics provide all the necessary information.
Step 2 Objectives
- Define the set of metrics to monitor and analyze the program.
Making It Happen
- Ensure your metrics cover the range of levels outlined above.
- Consider the needs of relevant stakeholders.
- Don’t worry about feasibility and cost of capturing data at this stage. You'll prioritize and focus your list of metrics later.
Up Next: Learning Evaluation Step 3, Discover
Our next blog post covers the third step in our seven steps of learning evaluation: discovering the effectiveness of your learning programs. Don't miss out and sign up to receive our latest blog posts.
About the author
As one of the authors of xAPI, Andrew Downes has years of expertise in data-driven learning design. With a background in instructional design and development, he’s well versed in creating learning experiences and platforms in corporate and academic environments.
Subscribe to our blog