MEASURING SUCCESS

How do you know if a training initiative is having the intended results? Most organizations are very good at collecting immediate and somewhat superficial data about training course results, but few go beyond this level of data collection or analysis, often believing that collecting more information will be too time-consuming and difficult, even though the data may have a major impact on the viability of the training initiative or the functional resources required to support it.

At Paul Terry Consulting Group, we help our clients gather data beyond the “did they like it” level. When presenting to senior leaders, even a little more data around the effectiveness of a training initiative can make a big difference as to whether the program gets further support. We present the levels of training evaluation (or measurement) below, using Donald Kirkpatrick’s and Jack Phillips’ work as a reference.

MEASUREMENT

How do you know if a training initiative is having the intended results? Most organizations are very good at collecting immediate and somewhat superficial data about training course results, but few go beyond this level of data collection or analysis, often believing that collecting more information will be too time-consuming and difficult, even though the data may have a major impact on the viability of the training initiative or the functional resources required to support it.

At Paul Terry Consulting Group, we help our clients gather data beyond the “did they like it” level. When presenting to senior leaders, even a little more data around the effectiveness of a training initiative can make a big difference as to whether the program gets further support. We present the levels of training evaluation (or measurement) below, using Donald Kirkpatrick’s and Jack Phillips’ work as a reference.

THE PROCESS

Level 1: Reaction / Satisfaction

Definition:

An evaluation typically completed at the end of the course on the overall reaction that participants have to the training both emotionally and cognitively. The evaluation focuses on the content of the training, the facilitator, and how the training was conducted.

Tools:

Most clients use a standard evaluation form to gather Level One data from all participants at the end of every workshop. The results for each item are often summarized on a scale, usually from 1 to 5. Some clients use a cut-off score (such as 4.0 on a 5.0 scale) to determine the value of the program.

Level 2: Learning / Cognitive Acquisition

Definition:

An evaluation of the extent to which the concepts, principles, knowledge, facts, processes, procedures, techniques, and skills that are presented in the training have been learned.

Tools:

Paul Terry Consulting helps clients develop a short pre/post survey (usually 10-15 items) to gather Level Two data. The pre survey is often administered at the start of the workshop. The exact same survey is then administered after the workshop, but it is labeled “post survey.” These results can then be tabulated and compared to show the increase in knowledge of the topic or increase in skill, as assessed by the participant.

Level 3: Application of Behaviors or Implementation on the Job

Definition:

An evaluation of behavioral change on the job, including the specific application of knowledge and skills that were learned in the training. Usually conducted 3-6 months after the training session. The power of the Level Three data collection lies in the examples shared–monetary savings, increased revenues, or other relevant measures used by the participants to demonstrate the value of the content.

Tools:

Paul Terry Consulting uses a library of survey items to gather Level Three data. We administer these surveys based on an assessment of the needs and requirements of the clients by mutual agreement. Usually 10-15 items are included along with 1-2 open-ended items to gather qualitative feedback or examples. The survey participants may include the training participant, his/her manager and others who can assess the application or behavior change. The data collected is at the individual or perhaps team level.

Level 4: Business Impact / Organization Outcomes

Definition:

An evaluation of the training’s influence or impact on organizational outcomes, most often measured quantitatively in such areas as quality, speed, cost, and revenue, but also measured qualitatively in areas such as customer satisfaction and retention. Obviously, it is difficult to isolate the impact of training on measurable business results apart from many other factors. For Level Four measurement to be successful, it is important that (1) success be defined up front in terms of future organizational outcomes, (2) that the appropriate Level Four measures be identified up front, and (3) that a plan is put in place to track these measures over time.

Tools:

Gathering Level Four data is a consultative process that is different for each client, depending on the business needs that have been identified. Level Four data are usually gathered by the client in the form of standard business metrics related to the areas of needs, such as those mentioned above. Paul Terry Consulting can help the client collect this data and analyze the results relative to the stated goals and objectives.

Level 5: Return on Investment

Definition:

An evaluation of the monetary value of the business impact (Level Four) compared to the cost of the training. This level of evaluation is the most impactful since it speaks directly to the investment outcome, just as for any other capital expenditure or other investment a client might make. However, few clients take their evaluation this far, unless the training is considered core to the organization’s business strategy.

Tools:

As with Level Four data, gathering Level Five data is a consultative process that is specific to the client. To achieve Level Five data requires that Level Four data are gathered first in order to make the calculation of return on investment (ROI). Once Level Four data are gathered, it is relatively simple to calculate ROI because the investment costs of implementing most training programs are easy to calculate.

READY TO GET STARTED?

Once the level of evaluation is determined, and the data are gathered, then the client has available very powerful information to reinforce the value of the program, and engage leadership in determining how to further support the implementation, and drive greater business results. When shared with the broader employee population, this information can also be very helpful in increasing awareness of the value of participating in or supporting the training initiative. Reference: The levels of data and definitions are taken largely from How to Measure Training Results, Jack J. Phillips and Ron Drew Stone, 2002, McGraw-Hill, New York.