Heard of Kirkpatrick? If you are in learning, you must have. How about Katzell?
Dr Raymond Katzell (1919 – 003) was an “industrial-organizational psychologist”, a major influence in the areas of motivation and job satisfaction. Among other things, his work dealt with “the effectiveness of productivity enhancement interventions.”
Our concepts of industry and organization have undergone a sea change from Dr Katzell’s times. Yet, today, are you likely to be influenced by such a scientist when you attempt to measure your learning? Dr Donald Kirkpatrick was. He drew on Dr Katzell’s teachings to develop the four-level model that remains popular and in use.
The Kirkpatrick Model
More than half a century ago, Dr Kirkpatrick thought like every L&D professional does (or ought to) today. He had “a genuine desire to know if his training was making a difference for the supervisors he trained, their direct reports, and the organizations in which they worked.” Not surprisingly,
Dr Kirkpatrick chose Evaluating Human Relations Programs for Industrial Foremen and Supervisors as the subject of his dissertation in 1954. And the four-level model was born.
Here is the model.
While many consider the model to be rather outdated to cope with the modern realities of business and learning, it has provided the foundation for all the models that followed.
Kaufman’s Model of Learning Evaluation
In 1994, Roger Kaufman and John M. Keller published what came to be known as Kaufman’s Five Levels of Evaluation, or more commonly, as Kaufman’s Model of Learning Evaluation.
Kaufman attempted to follow the already-established four levels of Kirkpatrick’s model. Hence the description of various levels as micro, macro and mega. However, as this can be rather confusing, many prefer to look at Kaufman’s work as a six-level model.
The Phillips ROI Model
Another expert to tweak the Kirkpatrick model was Jack Phillips, the founder of the ROI Institute. In the early 2000s, he added a level to the Kirkpatrick model.
Essentially a restatement of Kirkpatrick’s “Results” in monetary terms, ROI requires quantifiable business data before, during and after the learning program. One must also filter out non-training factors like seasonal effects, variations in the ecosystem and regulatory changes.
Brinkerhoff’s Success Case Method
Very few learning programs are exceptional successes or utter failures. Most fall somewhere between the two extremes. The Success Case Method (SCM) proposed by Prof Robert Brinkerhoff digs deep into the extreme success stories to learn why.
SCM is not really about the training program per se. After all, every training program is an event that provides an effective learning intervention. It is flanked by the first stage of creating focus and building intentionality and the third stage of supporting performance improvement.
SCM involves five key steps:
1. Planning a focused success case study.
2. Creating an “impact model” to define what success should look like.
3. Searching for the best and worst cases using a scientific survey.
4. Interviewing and documenting success cases.
5. Communicating findings, drawing conclusions, and proffering recommendations.
Learning-Transfer Evaluation Model (LTEM)
Developed by Dr Will Thalheimer, LTEM is also built on the Kirkpatrick foundation. Instead of four levels, LTEM has eight tiers.
This is what LTEM looks like.
Cut and shape your best-fit model
Trying to pick one from these is unlikely to help. Instead you would be better off, if you focus on the common attributes, the driving factors behind each model.
Dr Katzell wanted to assess motivation and job satisfaction. He inspired Dr Kirkpatrick who wanted to measure the impact on learners and the organization.
Kaufman separated the resources available from the quality of delivery and focused on the mega benefits delivered by learning. Phillips brought money to the table by emphasizing return on investment. Brinkerhoff took the empirical route to study the success stories and glean the learnings from those. Dr Thalheimer focused on the transfer of learning and its macro impact.
Learning is a must. But you are unlikely to borrow a program from the neighbor and run it. Why should it be any different when it comes to measuring.
Know your organization’s needs first. Structure your learning program accordingly. Given your business environment, the measure you need is the one that tells how far the learning has met business needs.
Earlier in this series:
