Here are the questions attendees asked Dwayne Thomas from Verizon during his live broadcast on April 23. Below are his answers.
For the promise of performance, you first need to have the knowledge, because knowledge is going to turn into skill, and skills into performance. So, just having the insight and understanding of where people are potentially tripping up with your content is a great way of knowing where there may be learning gaps. Obviously, if you pull an item-level analysis on a course or you just use something like Storyline to build a post-course exam, that item-level analysis is going to give you some insight into where there are still learning gaps.
I would encourage you to not always just use those instruments immediately after a training event, but continue to ask people questions over time, as there's a lot of science and evidence that says it re-engages their memory and recall. Try using some retention methods. Ask some questions immediately after, and then a week later, re-ask them the same questions but maybe a little bit differently. Dig into the item-level analysis to determine if there are gaps, and then determine if the gaps are in the training, in the way that we explained it, or in the way it is worded.
Honestly, it was her having a relentless pursuit of wanting to create the learning experience that she wanted. I really sold the fact that this is something for the future, and that we needed to do it now. Now is the time, and I'll let her be first. Even though we roll through the same senior president, it's a completely different executive vice president, so she was able to be first but through a partnership back to her peer’s organization. So, we provide a lot of their technology stuff. The selling point for her was she got what she wanted, and she also really got to be first in being able to deliver a new capability to the experiment, so it really wasn't a hard sale. When we can deliver anything other than from a technology perspective, we did.
We started with a single course. We knew that if we got involved in the procurement of an LRS, it would take awhile as it can take a long time for Verizon to buy stuff. We put the responsibility on the content vendor, Root, to partner with Watershed, so we would not be slowed by all of our processes and procedures. So, they actually pursued Watershed for a single course.
We used the single course as a proof of concept, and then we started our own contract to take over the agreement that Root had initiated with Watershed. We did it in full transparency—we didn't do it “wink wink” behind closed doors, but it was just the quickest way to market for us to solve the needs of the course being launched. It sounds like a really long time—it was like a four- or five-month development cycle—but it was for a change management course, and the content was still being baked and it wasn't a huge business need. It was just something that we knew we needed to put into the ecosystem to adjust some issues from an organizational development perspective.
I hate to admit it, but it's only Storyline so far. The first two courses were hand-coded HTML from the vendor, and the only other tool that we're using to communicate xAPI statements is Storyline 3 at this point. It seems to be more capable than Captivate in capturing the types of data that we want. With everything else we’ve seen and considered, it's a platform play where we're trying to integrate the learning platforms and have them make the xAPI calls.
I'll give you the pseudocode version of it. We're still somehow launching the course as a SCORM-enabled course. But there's an additional flag in the administrative setup that they know then to put a different lens on the course and to allow the xAPI statements to flow outside of that SCORM wrapper. So I don't know exactly how IT has accomplished it, but if necessary we can maybe connect and let you guys have a conversation.
I'm going to be honest. From that first conference, where I met Watershed, we had some very good detailed conversations over the course of two days. When I pick a vendor, I'm really looking for somebody who I think is going to make not a good vendor, but a good partner. And through the conversations and successive meetings that I had with Watershed, I just felt that with [Watershed], compared to the one or two others that I'd spoken to, I was going to get that unique partnership that was going to help drive me into the future—and they've absolutely delivered.
So, when I said that Root approached Watershed, they still created a unique partnership over that first course. They’ve been fantastic partners and have lived up to what I look for when I'm seeking a new relationship with the vendor.
Keep in mind, we've only got the two courses. First, the insights—because both of these courses are more learning experiences versus what you would consider a traditional course—they're not as much about the right and wrong answers, but about the reflection and the feeling that you have after the experience. They're not considered hardcore courses.
The other thing I would say, which I pointed out during the discussion, was apparently learners don't like really long videos—regardless of how good the videos are (I might be biased because I was involved in the development of them, and they're really compelling from a storytelling perspective, a graphic perspective, as well as a narration). Still, people time out, right? They're like “okay, time to move on.” So, that's one insight: be cautious in the length of your videos.
Beyond that, I would say it's interesting to see how few people actually will download resources after they go through the learning experience. The completion is much higher than those who are willing to download the resources to reuse them after the fact. And that's specific to the change management course, which was a second course that we developed. We will know more in about three months after we start scaling up our 50 or so Storyline developers, and they start leveraging xAPI and digging into the analytics in terms of what they can learn from what's happening behind the scenes perspective in the content.
We've not added any team members. We’re pausing a specific platform in order to both find and create a better learning experience. So we're stopping some work in order to put the resources onto the new work. Now, we're in this transition period where we're having to sweat a little bit more than usual to sustain both, but we're strategically leaving one in order to both find and create these new learning experiences.
When you look across the landscape—Verizon's roughly 150,000 employees and another 65,000 or so external partners in the LMS that we support—we have a pretty decent-sized learning technology team, and they’re all being affected by this pause, from the LMS administrator, to the exam and survey builders, to the reporting and analytics team. Basically, everybody is having to learn a new skill set and new capabilities. We don't have all the answers, but we're on this trajectory, this journey of taking the baby steps that you need to really to eat the elephant. It is a disruptive technology, but it's also one of the few things that I've truly gotten excited about in my 20 years in learning technology. It can be so disruptive and beneficial, but it's going to impact everything that we touch if we're going to deploy it and realize its full value. So, it’s more focus on upskill and skills than it is additional headcount.
The first two courses don't sit in the learning management system, but we've had a batch process where we’re manually updating and uploading completions once a week. So, it's not content that are forced on people that they have to take. So the timeliness isn’t as critical. The performance support tool is also sending this stream of data to Watershed in a batch process. I think it's hourly.
We're getting ready to do the video delivery platform. It's going to be sending xAPI calls and data consumption patterns to the LRS. So, we've not created a lot of individualized content just yet. Like I said, we just got security approval to move forward full steam ahead, but we've really kind of rationalized that in our customizations for the LMS. There are a lot of reasons why we would want to put content in our LMS. So, as part of the deliverables, the IT team has given us the ability to deep launch a course into the LMS and you never know till you touch the learning management system. Learners can type in a vanity URL, launch the content, close the content, and never know that they touched the learning management system. But, because that content has all the document retention policies on it— it's backed up and disaster recoverable—it makes sense for us to put the content in the LMS, but disguise the fact that you touch the learning management system. That's our intentional strategy for remaining legally compliant and having the horsepower capability of the content delivery platform while disguising the heavyweight launch that an LMS usually imposes on you.
Yes and no. Four years ago, when we were trying build the business case, we were looking at the open-source stuff and we could never get it over the goal line. It was too big of a commitment. You have to spend capital and dedicate resources to building this [open-source] environment internally—understanding the technology and the capability. And then like I said, the smart experiment in my case was to run a proof of concept with a provider, and if that was a failure we weren't going to continue. So, a minimal investment was needed to test the capabilities, the technology, the analytics, and the reporting and evaluate if the tool delivered on the promise before we made a larger investment.
Part of my strategy was, since we couldn’t get the capital infusion to do the open-source, was to set up the servers to collect the data. We're still a little timid around cloud-based solutions. Four years ago, cloud was not even in the conversation. So I know now that you can spin up all these open-source things and you're on your own virtual private cloud—but that wasn't available to us. It may be available to us now, but it definitely wasn't available to us four years. We do continually analyze it. We do think “is there a better way for us to do it” and we are trying to figure out “is there a way for us to bring that data into Qlik, which is something that we're starting to move to strategically from a BI perspective.
No, but with our capabilities that we're getting with xAPI there's no reason why we couldn't build a really lightweight exam that would be mobile and be able to bring in people's participation and content during a live event. Virtually everybody has a cell phone now. And while it's a little complicated for us to bring people into an exam situation on a mobile device with what we’re delivering next month in our current ecosystem, we should have a much better learning experience and be capable of capturing content—whether it's survey or exam related during or immediately after a live lecture.
I'll go back to my last slide. Think about that small thing and whether it's understanding a stakeholder and their needs and what their pain points are, or something that you know specifically that you can pursue to gain the rallying cry of a group of people. Find out one little thing and figure out how to position it in front of them so you can say “well we can do that, but it's going to take some latitude.” Whether it's spinning up the free version of Watershed that Andrew mentioned, just to do a proof of concept, or engage with them for maybe something a little bit bigger and audacious. Try to set up a smart experiment, that's where I would absolutely start because it's all going to take a long time for you to fully implement all the advantages that we shared on those mountain peaks.
The xAPI xAPRIL page will be updated with loads of case studies, prizes, and tools throughout the month of April. Make sure to bookmark it and check back often.