Measurement in the Age of COVID-19

COVID-19 has caused some organizations to fundamentally change how they deliver learning, while in others, it has accelerated a change already under way. So, the question naturally arises, “Does measurement need to change as well?”

As you might guess, the answer is both yes and no. Let’ address the “no” first. The fundamentals of measurement have not changed. You should still capture all the relevant activity or level 0 metrics like number of participants, courses, completions rates and dates, and costs, to name the most common ones. You should measure level 1 participant reaction, and you should measure the amount of learning (level 2) where that is appropriate. If the course is part of a strategic initiative (like increasing sales or improving quality), then you should also measure level 3 application and level 4 impact. You may also wish to compute the ROI (level).

The “yes” part brings us to some measures specific to the transformation to online learning, which - while important prior to the pandemic - are even more important now. Here I am thinking of measures like the percentage of instructor-led training (ILT) converted to either web-based (WBT or online) or virtual ILT (synchronous). I bet many of you have tracked these since March and hopefully by now are about where you hoped to be.

Of course, we all know that in an emergency, the quality of the transformed learning may not be high. Many simply put PowerPoints from ILT online. In other words, the learning was not designed for online and thus was not really “transformed”. You should measure the impact of this by comparing your level 1, 2, and 3 scores pre and post March. How much has your quality suffered? Has the application rate dropped and the scrap rate increased? (Scrap rate is 1 – applications rate) What are your plans for truly transforming ILT to WBT? You will need a measure to track the percentage of ILT content purposely redesigned for WBT along with measures of how long it takes and the level of effort required so you can better budget for the future.

In most organizations, there is also heightened attention to utilization of your online catalogs, portal content, and knowledge sharing. Have you captured this increase through measures like percentage of available courses utilized, percentage of employees taking one or more courses, percentage of portal content accessed, percentage of employees using the portal, percentage of employees who are actively engaged in a community of practice, and number of active communities? These metrics will help you tell a good story about the opportunities provided by L&D during COVID-19 and help build your business case for greater investment in these areas.

Bottom line, you have worked incredibly hard to transform learning in a very short period of time so your organization can carry on remotely. Be sure to make a little time to capture the measures you need to position yourself for further success and to let others know how important learning has been in this challenging time.

Learn more about the impact of COVID-19 on learning and about measurement in general at the upcoming Center for Talent Reporting Annual Conference October 27-29. Skillsoft’s Laura Rexford, vice president of sales effectiveness, will be a speaker at this year’s free and virtual conference.

David Vance, Executive Director, Center for Talent Reporting