By Lisa Harrison
When I think about measuring Return on Investment (ROI) of a training program, my brain lodges a protest. I agree that we need to measure the benefits of training – I just don’t agree that ROI is an appropriate measure to use. ROI is far too simple to cope with the complexity of professional development and performance management.
I suggest that what we really want to measure is the benefit vs cost, or ‘Cost-Benefit Analysis’.
So in discussing ‘ROI’ on learning, I’ll refer to it as measuring the benefits of learning.
We want to be sure that our efforts in professional and/or personal development are benefitting the staff and the organisation.
The reasons we send people to learning programs include, improving individual performance and improving how that performance supports the organisation’s ability to meet its operational and strategic goals.
Specifically, training or learning is implemented because there is a ‘need’. It may be the entire organisation, a team, or a few people. It may be a need for a current role or future potential. It may be to increase knowledge, improve skill or change attitude.
So to measure the effectiveness of the learning, measure the need. And that’s why Kirkpatrick’s Levels of Evaluation are so popular in Learning and Development. They certainly enable us to measure the benefits.
Donald Kirkpatrick identified four levels of evaluation. As you move up the levels, each becomes more aligned with business outcomes, more clearly measuring the actual learning, but also more expensive and more resource-intensive to administer. And like most things in life, you get what you pay for.
Evaluation at this level simply involves asking learners for their opinion of the program. This level has now been so disrespected that the surveys are referred to as ‘happy sheets’. And there are some good reasons why.
In 18 years of training, I’ve commonly seen happy sheets that gave me an unexpected insight into the ‘burning issues’ for participants, i.e. the quality of the food and coffee supplied, the temperature or décor in the training room, etc.
This was obviously front of mind for the participants far more than the performance improvement they experienced. And with people keen to get out, catch the next train, read the 427 emails waiting in their inbox, it’s no wonder the happy sheets can get the short shrift.
With Level 1 being simple and cheap, everyone at least does that much to measure training, with the benefit of knowing to buy the muffins and coffee from a different café next time. But to be honest, the evaluations are valuable feedback for those who design and deliver the program so it’s an important step.
This level is much more exciting because we are actually measuring (assessing) the learning. We conduct assessments to see what people know and know how to do. The catch with level 2 is that you need to be careful of what you are measuring.
At the end of a Customer Service program, I can set an exam with questions about conflict management, but that only shows me how the participants would theoretically handle an angry customer. It’s much more robust to observe them actually dealing with the situation in a simulation or (even better), a real live frothy-mouthed customer!
So Level 2 could be slightly more or a lot more expensive, depending on what you are assessing.
Knowledge recall, for example, might be a reasonably cheap exam-style assessment. But usually in workplace learning, it’s the application of the knowledge, the skills and the attitude that we really want to measure, which can be a large time and cost investment for assessors observing individuals.
The disadvantage here is that you only measure the change in the individual, not how that affects the business environment and bottom line.
This level measures the application of the learning in the workplace and real life situations, not just in a training room simulation.
This is where we start to look at embedding skills and ongoing and sustainable behaviour change. Or rather, the performance improvement desired when spending money on training.
How do we measure this?
This is the real outcome from the learning as it applies to the individual. You are measuring the change in that employee.
Performance reviews, manager feedback, colleague feedback, and all of the performance management processes that (hopefully) your organisation uses for staff development will assist in measuring the individual’s performance and behaviour changes.
Whilst these measurements can be time and resource intensive, if they are already in place as part of your staff development, they are worthwhile maintaining and keeping up-to-date.
This level gets to the real heart of business strategies by measuring the benefit to the business, not just the individual.
If it’s in the example of customer service, you may ask whether there has been a reduction in the customer attrition rate or an increase in the Net Promoter Score. The measures that you already use for your business will be (or should be) covering this.
If the objective of the learning program is aligned to the business objectives, then you are likely to already have measures in place.
In summary, if you want to measure ‘ROI’: Level 3 is where you will see the ‘ROI’ for the individual, and Level 4 is where you will see the ‘ROI’ for the organisation.
If you’re measuring results, you’re measuring benefit.
If you don’t have a baseline to measure against, you won’t know whether the learning program actually influenced the figures. So measuring before the learning program will give you a clear benchmark to measure against.
If the learning outcomes from the program don’t support the business outcomes, not surprisingly, a fantastic result at Level 2 won’t translate into a correspondingly brilliant Level 4.
Be sure that the identified learning need is what needs to be improved. Delve deeper. Fix the need, not the symptom. In other words, make sure your learning program is the right tool for the job it needs to do.
With a bit of pre-planning and goal setting, you can achieve far more from your learning intervention, and measure the benefits of investing in your people. The return on investment from training is a more engaged, capable and productive workforce whose performance directly impacts the organisation’s ability to meet strategic goals.