We've explored the world of learning analytics, including the ins and outs of a learning analytics platform. Now, it's time to hear from several learning and development (L&D) practitioners—who will share their professional insights and advice for creating and maintaining world-class learning programs. We kick off this portion of our series with Sam Worobec, who served as the director of training and internal communications at Chipotle Mexican Grill.
What’s your data capture strategy for tracking learning experiences, and why did you choose it?
Our primary format is xAPI because of all of the information that comes with it, and the fact that it is more HMTL5 friendly than SCORM. We have mobilized our learning platform to tablets and phones, and our content needed to shift along with it. xAPI was an easy transition for our eLearning Developers as well as our platform vendors to adopt.
Our secondary format is the litany of business performance metrics within our organization. By tying in our learning experiences with performance metrics, we are able to see not just what people are learning, but also how they are performing back on the job.
How do you decide when you have enough data (or too much data)? Do you have suggestions for detecting and cleaning up bad data?
You can never have too much data. The challenge is getting the right data. Once you start digging in and looking at the data you do have, you’ll realize what questions you should be asking, what data you really need, and how it needs to be formatted going forward.
You can never have too much data. The challenge is getting the right data.
This will be your guiding light as you then work to clean up any data that would be useful, but isn’t currently useable because of where it is located or how it is formatted. The key, though, is to not focus on the data, but rather what questions you have that you need data to answer.
Knowing exactly what data to look at and its meaning can be challenging. What are some recommendations or examples for learning benchmarks and KPIs?
Most learning organizations will start with Kirpatrick’s four levels of evaluation. They then begin to get their Level 1 data and determine if people even like the training. Some get to Level 2, where they analyze what people learned during the training by measuring knowledge before and after the training. This is where most learning organizations spend most of their time, and where they often stop.
My advice is to start at Level 4 rather than Level 1. Every piece of learning that is being created should have some sort of a business result in mind. Ask yourself, “What is the business impact of this piece of learning?” and “What is supposed to happen once someone takes this training?”. Once you know that, you can identify what behaviors are needed to accomplish that business impact. Then, determine what people need to learn in order to enable that behavior. Finally, ask them if they liked the training.
In this model, you can measure the business impact, behavior change, learning, and enjoyment all at the same time—and the meaning to each piece of learning is built in from the start.
What are a few small ways people can regularly improve their learning?
Learning programs are only effective if people attend the program, then follow through on the behaviors identified in the program. While you may or may not be creating new training every week, you can be gathering either quantitative or qualitative data to understand why people are or are not taking the training and, more important, why they are or are not applying what they learned.
Knowing the answers to these questions will inform what you need to alter about your existing programs and what you need to keep in mind about future programs. The best part, is that this can be as simple as going out into the business and talking to your participants to really understand what is happening.
There are many metrics to choose from when it comes to showing ROI. How do you prove the value of learning in your organization, and which metrics do you consider to be the most powerful?
The two measures that I start with are always the main business drivers of the organization and the retention of employees. After attending my training program, are participants more or less likely to achieve the business goals of the company (which are unique to each company), and are they more or less likely to stay with the organization?
If I can create a training system where participants are more likely to achieve the business goals of the company and are more likely to stay with the company, I have created undeniable business value for the company.
Sometimes the boss is the only obstacle standing in the way of doing great things. What are your suggestions for getting organizational buy-in with data?
The first suggestion is to understand what the boss is trying to achieve. Tying data to your boss’s goals is an easy way to get buy-in to the data. What are her goals and what are the things getting in the way of her goals? If you can understand what she is trying to accomplish, and if you have an idea of how she can accomplish it, data will be the key to proving your idea has merit. Without data, your ideas are just ideas. They are subjective and may be overlooked. Data provides an objectivity that trumps an idea every time.
Without data, your ideas are just ideas. They are subjective and may be overlooked. Data provides an objectivity that trumps an idea every time.
When the data you are providing has no obvious business value, or doesn’t have enough value to garner the attention of your boss, look at the bigger picture of your organization to understand how your “great idea” fits in to the larger scope of the company. If you can tie the results of your great idea to something of benefit for the company, you may have a shot at getting buy-in.
What are your goals or future plans for your learning analytics program?
I have two primary goals for analytics in the future. The first is to combine learning analytics with an adaptive learning framework to automatically trigger the best learning path for an individual employee. The adaptivity will be based not only on what employees need to know and which way they learn best, but also based on the learning paths of the most successful employees in the company. In this way, each learning will be provided with the learning path that is both best for an employee and for the company.
The second goal is to specify business outcomes down to the individual level, then tie those outcomes to learning in order to truly tie business KPIs and on-the-job behavior to learning outcomes. An example of this would be any type of audit that occurs in your business (i.e., customer satisfaction survey or an operations audit). Rather than looking at unit level results or just holding a general manager responsible for the numbers of the entire unit, my hope is to understand who exactly was working at the time of the audit and track the results of that individual audit to the individuals that were there at the time of the audit.
If remedial training is required, an adaptive system would automatically trigger remedial training for those employees that need it. If the team should be rewarded for an outstanding audit, the adaptive system would trigger rewards for those employees working. With this in place, it will be far easier to track performance and behavior at an individual level, and move corporations away from an audit and punishment culture and into a more individual merit-based reward system.
L&D Spotlight: Applied Industrial Solutions
Next, we'll talk with Andy Webb, a talented L&D professional who knows how to develop strategies and solutions linked to growth, performance improvement, and business goals. Be sure to sign up for our blog to have the next post sent straight to your inbox.
About the author
Starting as a content developer, Sam Worobec has created FDIC compliance training, written leadership development for Fortune 50 companies, created training for a global renewable energy corporation, and took Chipotle to digital and mobile training platforms that engaged more than 70,000 employees.
Subscribe to our blog