<img height="1" width="1" style="display:none;" alt="" src="https://dc.ads.linkedin.com/collect/?pid=114414&amp;fmt=gif">

xAPI Spotlight 1:  Getting Data Out


Behr-xAPI

SPEAKER
Brian Floyd, The Behr Paint Company
Director Sales Training Development


THINK xAPI ISN'T GOOD FOR REPORTING OUTSIDE AN LRS?

Hear how The Behr Paint Company uses xAPI to feed data into other applications that power sales training and performance support. 

 

Don't have time to watch the recording? Check out the written recap

Q&A with Brian Floyd from Behr

Here are the questions attendees asked Brian during his live broadcast on April 13. Although he ran out of time to address every question during the session, we've provided his answers to everything below.

  1. In terms of interoperability, what other programs can xAPI work with? Can it work with email programs? Chat tools? Performance Management tools?

  2. Is all of your content managed or stored in a single site or LMS?

  3. What LRS do you use and is it integrated with the LMS?

  4. How successful have you been with tying content trending to business KPIs to validate that your approach is really making a difference to your business?

  5. What changes to your team or organization roles resulted from implementing and supporting xAPI and the LRS?

  6. Are you capturing business results in the LRS?

  7. Thank you again. What qualities does your xAPI wizard have?

  8. What, if any, are the limitations of xAPI from your experience?

  9. How is xAPI affected by GDPR?

  10. Who do you use for your LRS?

  11. Is the content in the app pulled from the LMS? Or is it standalone?

  12. I'm excited you are using captivate. Are you getting xAPI data from those program? How?

  13. Thanks for letting dream bigger than I was. Amazing. What do you see as "running"?

  14. Did your internal IT department build the app initially and the reboot of it?  Or did you have an external vendor build it?

  15. How did you get started with learning the capabilities and uses for xAPI?

  16. How did you scale the launch of this initiative?  Did you run a pilot? If so, how did you choose the pilot group?

  17. Does someone in your L&D department code the API statements or do you partner with IT to do that?

  18. I may have missed this, but how do users access the app if they aren't on an iPhone or iPad?

  19. Did you come up with your own verb and object taxonomies for a lot of the different user actions that you are tracking in the app and microsites? Thanks.



1. In terms of interoperability, what other programs can xAPI work with? Can it work with email programs? Chat tools? Performance Management tools?

This is what I find most exciting about xAPI—any experience the user is having is ultimately able to be captured.  

The question isn’t can they work with xAPI, the bigger question is can you get the data out of them? Many pieces of software have an API (a way to ask software for information) to let you do just that, which can be then translated into an xAPI statement. We are mainly using our custom-built solutions so we haven’t explored the interoperability of the specification.

I have one business use case, and one I did for fun. We connect to polleverywhere through an API that allows us to record our sessions via xAPI to our LRS. We also switched from Survey Monkey to Survey Gizmo because it [Survey Gizmo] reports to our LRS right out of the box. The other, is my son wrote an API that asks Spotify what song he is listening to on his app so he can display it on his website, I turned that into an xAPI statement reporting into my personal essentials account that I have with watershed. The only purpose was for fun and exploration. I also now question some of his music tastes :)

2. Is all of your content managed or stored in a single site or LMS?

Nothing we have from a sales training perspective still lives on an LMS. Our HR department still uses one for compliance training. The majority of our content lives on our app server, our web server, or is linking to the Behr.com and a DAM (Digital Asset Manager) that our company has.

3. What LRS do you use and is it integrated with the LMS?

Watershed is our LRS, we have not connected to our LMS (see answer above). We have reached out to HR to see if they are interested in joining us in exporting the data from LMS into LRS, but they haven’t pursued this option to date.

4. How successful have you been with tying content trending to business KPIs to validate your approach is really making a difference to your business?

On a a scale of 1 - 10, I would give us a solid 6—probably the most significant was tying the training and curriculum we produced to performance that our reps exhibited when secret shoppers from JD Power engage them. We saw significant improvement in every one of the six key metrics that we measured, and it coincided brilliantly with the consumption and usage of content. We also were able to make some great business decisions about this training. The first round of training we did live situational videos with actors, the second round we did animated videos using goAnimate.

We found that 3 minutes was the sweet spot for video length, and that the field and their customer far preferred the fun and whimsical goAnimate characters to that of real actors. xAPI and our LRS, coupled with the market research data, made these decisions easy, and fact based rather than gut feel.

We used to fly a team of people together to produce these actor training videos. Analytics proved that it could be done remotely just capturing voiceovers and marrying that to our avatar counterparts. This saved the organization tens of thousands in travel and production costs, and xAPI showed it was a more widely adopted and effective form of delivery.

5. What changes to your team or organization roles resulted from implementing and supporting xAPI and the LRS?

We have really only had one significant organizational change. We now have an app manager/xAPI analyst that handles all of the data in the LRS and is has become an expert at navigating the LRS to produce the data needed to make business cases and test theoretical questions.

The biggest change really came in our mindset. When our team develops content, they do say they’re now consciously considering what could be observed in the experience.  

6. Are you capturing business results in the LRS?

Not yet, that is when I think we hit the "run" phase of the crawl, walk, run strategy. I will tell you, though, we do export a fair bit of data from our LRS into Excel, and our analyst (who I refer to affectionately as my excel sorceress) combines results and trends with sales data and market research data. Our long-range plan is to report this into the LRS and automate some of this work

7. Thank you again. What qualities does your xAPI wizard have?

The number one quality I would say is an insatiable appetite to learn. She strives to understand and learn. She is also super organized and has an analytic mind. When you are fortunate enough to have employees like these, give them a long leash. She surprises me regularly with new and interesting capabilities because she is allowed and encouraged to think big, take risks, and fail.

I wouldn’t say she is an xAPI wizard as much as a data wizard. Once in a while, she digs into the statements, or asks questions - but the majority of her time is spent finding correlations between the data and behaviors. Our LRS provides the data filtering, reportingand visualization - she marries that information with our initiatives and goals.

Once xAPI is up and running, the heavy lifting is done—it just goes into reporting mode like everything else. SQL runs a huge amount of data that we all look at every day, yet few of us could write one SQL query if we had too. xAPI is very similar, once the statements are reporting in they fade into the background, and the data comes the the forefront.

8. What, if any, are the limitations of xAPI from your experience?

There are limitations, but I don’t believe they are xAPI’s limitations. Rather the limitations lie on whether you are able to get the data out. How do you record ILT (Instructor Led Training) with xAPI? If you have the data, any information about any experience can be fed into the LRS. For instance, we use Webex to host our webinars. Webex does not have an API yet to get the data out directly, so we capture attendance manually and then import it into our LRS using CSV (Comma Separated Value) report that we generate from Webex. So, the limitation is how hard do you want to work on the front end to capture the data—what is practical? We work daily to find and incorporate solutions that automate all of this data reporting moving forward.

9. How is xAPI affected by GDPR?

Answer provided by Watershed: xAPI as a specification isn't really impacted by GDPR, although GDPR can affect how it's implemented in certain situations. For example, implementers are encouraged to use the HTTPS-only standard for secure communication—but that is an implementation recommendation, not an element of the actual xAPI technical specification. All technology providers, whether xAPI or not, need comprehensive information security policies and procedures, as well as application functionality to support the rights of data subjects. It is also important for Learning Record Providers and Learning Record Stores to understand their relationships with their clients to establish controller/processor relationships and data processing addendums/model clauses where necessary.

10. Who do you use for your LRS?

Watershed—a great team of folks to work with!  But don’t think to start your journey you have to make a full investment. We have a Watershed paid solution that we use for Behr, as well as a sandbox environment that we use for testing.

I, personally, also have a SCORM cloud LRS account and a Watershed Essentials LRS personal account (free). These free accounts have basic reporting information, and allow you to send and query statements and allow you to tinker without making a commitment.

11. Is the content in the app pulled from the LMS? Or is it stand alone?

Our app is powered by Appcelerator which is a hosting company for apps. The content lives in our internal Behr server and Appcelerator pulls and serves up the content as the app requests it. This makes the content available from either the app interface, or we provide a link to the content that pulls it to any web browser directly from our servers.

12. I'm excited you are using captivate. Are you getting xAPI data from those program? How?

First - Captivate does have some xAPI interoperability right out of the box. All of the playing we did with it though, seemed to be connected with being able to get user info out of the LMS.

Here is how we do it: Captivate generates HTML5, javascript, and CSS to produce the learning. We add in our own custom script to each piece that becomes available for captivate to call. In some instances, our app or other access methods are passing the user information though the URL the user is accessing and our custom script "catches that." In some instances, we have a Captivate screen that asks for users’ email addresses, looks up those users in our LRS, and admits them to the learning based on being existing users in the LMS.

The statement we want to send is hard coded in javascript, but can be passed parameters. We then use the javascript console in Captivate to access this. For instance, let’s say that we want the user to review a web page. When they click that button we use Captivate to open the content in the web browser and in the javascript console we pass - access(URL).  

On the backside, we have a javascript file we added that has access a function. What this does in Captivate is call the javascript function and passes the variable url (which is the web page) and our script reports the statement "User accessed google.com" to the LRS.

I know what I describe may sound daunting, but a mediocre javascript coder can accomplish this easily (I know because I am mediocre at best, and I wrote our script). It took me about 20 hours having no idea what I was doing and just trying stuff and googling stuff until it worked. I have since refined that script, and I use the same one over and over again for each captivate piece and just call different verb functions I have created to report.  

If you are interested, I am more than happy to do a crash course on "Hacking Captivate" to play with xAPI. Email me at brian@brianfloyd.me and I will set up a session to show you (and anyone else interested) the process.

13. Thanks for letting dream bigger than I was. Amazing. What do you see as running?
Captivate and limitation - yes. Thanks for telling me it had to be coded. Do you have a coder or was it added on to another person's job like an instructional designer?

See question above. For Captivate specifically, I coded the solution that fit our need. We have a code team that develops and maintains our app, but we do not have a full-time coder just for xAPI as you mention, like an instructional designer. I am adding such a person to my 2019 budget request though so we can continue to add more and more components that report into xAPI.

I believe the learning designer of the future will have to know coding basics, and that will become a job requirement.  

Creating the foundation to do this can take a little time and effort, but once it has been built it is just reused over and over again for each captivate learning piece you create.

14. Did your internal IT department build the app initially and the reboot of it?  Or did you have an external vendor build it?

An external app company built the pilot app that we repurposed. The redesign of our app was a combination of one external iOS coding contractor, an internal software engineer, and an internal Project Manager, and QA team. I will address the elephant in the room...How much did this cost us? I can’t speak to how much the pilot app cost to build, if I had to guess based on experience it likely had a price tag of about $20k. When we put out an RFP for the rebuild, our costs landed at $60K. It was $20K for the UI (which was great for the start, but we do our own UI design now), and $40K for the external coder and about a six-month build time.

We have since made some significant improvements, and over the last three years have spent around $60K a year average out to maintain this using an outside coder. Late last year, we added a full-time coder to our India team whose sole responsibility is the code base for our app.

15. How did you get started with learning the capabilities and uses for xAPI?

Events like this [xAPI xAPRIL], lots of googling, and reading, and a determination to harness what could be done. The good news is the xAPI community is small, but growing every day. I haven’t met anyone yet that wasn’t willing to help and share their experience. There are all kinds of new events and things going on. One of the coolest I have found recently is Torrance Learnings xAPI cohorts. I, unfortunately, found the spring cohort late this year, and have been more a lurker than a contributor, but I am already excited to join the fall cohort. These cohorts are packed with coders, instructional designers, dreamers, and newcomers. Every aspect of practitioner you needed is represented in these groups.

16. How did you scale the launch of this initiative?  Did you run a pilot? If so, how did you choose the pilot group?

Because we got the initial app at no cost and all of our reps had devices, we released it to everyone. We were able to learn enough from this to put together the RFP for the app I demonstrated.

I would choose a pilot group based on a good mix of early adopters, early and late majority adopters with a healthy sprinkling of your laggards as a reality check.

17. Does someone in your L&D department code the API statements or do you partner with IT to do that?

We do it (mainly I do it - but I am working to bring others up to speed). JSON statement are how xAPI expresses itself. JSON is not really "coding" per say. It is just a group of key value pairs. The xAPI syntax or recipes are already defined, so you just have to learn to read it This just takes practice, JSON looks like gobbledygook—but if you approach it a different way and treat it almost like learning a new language (but way easier!) and understand that with practice over time—it becomes natural. Really, anyone can write the statement.

The second part to the answer is once you write one a statement that fits your environment, you really aren’t rewriting anything. You are simply passing in different values of the statement, or perhaps adding another key.

18. I may have missed this, but how do users access the app if they aren't on an iPhone or iPad?

Sorry I could have easily spoken for two hours, and I know I moved at a lightning pace. I wanted to provide enough context so everyone could understand the impact xAPI can have.

The primary way of accessing is through the app, but any of the content in the app can be shared and forwarded as a "universal link." These links, when sent to another app user, will open the content in the app itself, but if you are not part of our app ecosystem it opens in any standard web browser.

Currently, we do not have a web interface to search the content directly without the app, but we are working on building a web app light version that we aim to deploy by year end.

19. Did you come up with your own verb and object taxonomies for a lot of the different user actions that you are tracking in the app and microsites? Thanks.

We spend a lot of time on "the registry" looking for the right verb and object types that are already defined. There have been cases, though, where we definitely have created our own. “Scanned” for instance is one of those verbs we made up. When we track what our users are scanning when searching for content this is the verb where using. We have also made up several extensions to support additional data we are capturing.  

For instance, we have a mini app in our app that converts the formulation of paints from one size to another. We capture this conversion in the extension in the statement so that we can store and recall the converted formulas as a later time. Definitely unique to our business, so we couldn’t find an extension context that fit so we made up our own.

Anytime a new reporting request or challenge comes up, though, I always start here: https://registry.tincanapi.com/# This registry is great for inspiration as well, specifically in imagining what could I be generating statements with.


Return to xAPI xAPRIL