It might not always feel like it, but not all learning happens online. In industries such as healthcare, construction, retail, hospitality, and travel, work happens in the real world. And while advances in virtual reality (VR) are exciting, practical skills are often best developed with hands-on training.
Observation checklist applications let you record these real-world experiences, and knowing how to capture and aggregate that data is critical for comprehensive L&D reporting. This blog post discusses the data requirements you need to assess and report on skills and provide feedback to employees.
What’s an observation checklist application, and how does it typically fit into a learning ecosystem?
Observation checklist apps (which usually run on mobile devices) provide interactive webforms to structure and systematize observations of either a training simulation or completion of a real job task. Using these forms, observers can capture data about how well employees completed tasks.
This checklist was created with Xapify, a skills development platform, and ensures medical professionals take appropriate infection control precautions.
A checklist application may be a stand-alone product or may form part of a broader skills management platform. A skills management platform collects information about learner skills and supports the development of additional skills through learning plans. The observational checklist application is used to assess skill competency.
Data may include a combination of:
- rating scores against performance metrics, such as product knowledge
- a record of process steps completed or missed
- errors selected from a list of common mistakes
- the time taken to complete specific tasks (this is especially relevant in a medical context)
- comments from the observer
This screenshot shows how you might visualize data from a checklist to identify which competencies are (and are not) most commonly demonstrated during observations.
Depending on the specific features of the checklist app, you may or may not be able to integrate it with other platforms in your learning ecosystem.
For example, Xapify is a skills development platform that includes an observational checklist. Xapify can assign learning content to employees based on any knowledge or skills gaps identified during observations. As a result, Xapify requires integrations with all the systems in which that training content resides. Other L&D observational checklist apps that do not have this functionality may not integrate with other platforms, aside from data extraction to the learning analytics platform.
Observation Checklist Software Data Requirements and What to Ask Your Vendor
What data should I look to pull from my checklist application for reporting in the LAP?
As you might expect, the most critical data to capture is the data generated from answers to checklist questions. This data should include identifiers for:
- both the employee and the observer,
- the question, and
- the observer’s response.
What does the data tell you, or what kind of things can you learn from it?
This data can be used at an individual employee level to see how someone performed a particular observed task. For example, you might use these reports to:
- Review with the employee as part of a coaching discussion.
- Inform decisions to assign training to the employee.
- Inform the employee’s manager in relation to the employee’s performance.
This Watershed dashboard presents some example data generated by Xapify using a checklist. It shows scores against three competencies by learner, average scores by competency, and comments made by the observer.
The data also can be aggregated to give a view of competency and skills at a higher level. For example, you might use these reports to:
- Identify exceptionally high-performing employees to take advantage of their expertise or to inform reward and recognition schemes.
- Identify particularly low-performing employees to provide support or inform other decisions.
- Map levels of skill proficiency and skill gaps across the organizations.
- Inform decisions about the training required to address commonly identified issues.
What questions should I ask my vendor about getting the data out?
The purpose of observational checklist apps is data collection, so you should expect apps you are looking at to have a variety of robust and well-documented options for data extraction.
Ideally, the app will have a good xAPI implementation (which is a particular selling point of Xapify). At a minimum, you should expect CSV data to be available via API. When considering a checklist app for workplace observations, ask the vendor if they:
- Have experience integrating with an LRS or LAP.
- In particular, have they integrated with your LAP?
- Support xAPI tracking of checklists.
- If not, do they support extracting the data as CSVs via API?
- Can they provide the documentation?
What if I already have an observational checklist app?
Once you are using a checklist app, there are some key considerations to ensure the quality of the data generated:
- Ensure the employee and observer identifiers match (or are mappable to) the identifiers used in your other data.
- When editing checklists, make sure that you understand your revisions will impact the data. Discuss this with your LAP and checklist app vendors and run tests for yourself if you are unsure how edits will affect the data.
- When creating new checklists, review them carefully and consider piloting them before using them more widely. This way, you avoid having to make future changes that might impact the data.
If you edit a previously used checklist, you may run into issues with inconsistent data. For example, if you replace a question, you may not be able to compare the data from the original question with the data relating to the updated one.
Observation Checklist Applications and Your Learning Ecosystem
What are the benefits of combining checklist data with other tools and systems within my ecosystem?
You can use observational checklist data alongside data from other systems to compare employee performance. For instance, do employees who complete training perform better during observations than those who don’t complete training?
This information enables you to identify which training best supports good performance. Conversely, it also shows which training does not contribute to performance (or might even be harmful). By combining the data sets, you will be able to ask these kinds of questions:
- Which learners significantly improved between observations, and what learning did they complete in between? This information will help identify particularly impactful training, which can then inform the design and curation of new training.
- What digital credentials and skills do people who perform well in observations have compared to those who perform poorly? This information will help inform decisions about hiring, internal promotion, and training.
- Where are there gaps in training that either lead to or fail to prevent errors? This information will enable you to create, curate, or promote training to address these gaps.
The checklist data needs to be mappable to the rest of your learning data for this to work. That’s why the observational checklist app needs to use a mappable learner identifier.
What about future changes to the ecosystem—how will adding, replacing, or removing specific tools affect checklist data, or will it?
Changes to the ecosystem are unlikely to affect checklist data.
The Role of Observation Checklist Apps in Skills and Compliance
Is a checklist app a good system of record for compliance reporting?
Where proper completion of tasks and processes is a compliance requirement (e.g. in healthcare), data from the checklist app can be a crucial input to compliance reporting. This scenario is particularly common in healthcare settings.
For example, one Watershed client uses Xapify to ensure healthcare providers wash their hands at appropriate times. This data informs training needs and also may be relevant for compliance.
How does a learning analytics platform help with skills data generated in an observational checklist app?
Three elements that can contribute to evidencing that a person has a skill:
- Evidence that the person has put effort into developing that skill, such as a record of learning completed or hours of practice logged
- Evidence that the person has mastered the skill, such as successful completion of an assessment
- Evidence that the person has applied the skill in a real-world context, such as within a work project
Successful completion of observed tasks can provide good evidence of applied skills (and could even be used to inform the awarding of digital credentials). Use checklist data to report on applied skills by mapping skills to observations or parts of observations.
Up Next: HRIS Data Requirements for Learning Analytics
So far in this series, we’ve looked at platforms and applications that generate data relating to learning activities. But learning data can be much more interesting and useful when combined with data about learners from an HRIS system.
This data means you can compare the learning activity of different parts of the organization or restrict the data shown in a report to a particular department. In the next post, we’ll explore data requirements for HRIS data—including how to get that data into your LAP and how you can use that data. Be sure to scroll up and subscribe to Watershed Insights so you don’t miss out!
About the author
As one of the authors of xAPI, Andrew Downes has years of expertise in data-driven learning design. With a background in instructional design and development, he’s well versed in creating learning experiences and platforms in corporate and academic environments.
Subscribe to our blog