Omega Services

SIX SIMPLE WAYS TO EVALUATE MANAGEMENT TRAINING

<< Back to News

By Michael Lee Smith, SPHR

May 2001
Reviewed August 2002

Evaluating management training is infrequently done, even though a great deal of time and money is devoted to the planning and delivery of the management training. It is essential to provide reinforcement to training course "alumni" to ensure transfer of learning. It would be ideal to watch the participants as they try out new behavior and give them some friendly free advice and encouragement. Unfortunately in many, if not most, instances there is little opportunity by the training staff to do this. I have developed a way to deal with this dilemma that may not please the purists but does fit within the constraints we face.

I was part of a small training staff located at the Northeast headquarters of a diversified company engaged in providing environmental and consulting engineering services. We were responsible for delivering management training to eight affiliated companies in thirteen locations. Since it is more cost-effective for the trainers to travel, we conducted a class at a field location and departed shortly thereafter. Except for telephone contacts, there was little opportunity to follow-up transfer of training. How do we know the training was effective? We conducted follow-up evaluations.

FOLLOW-UP METHODS

We used six types of follow-up. Which method we used depended on the content of the course, the utility of the method for measuring change and for motivating change and lastly, the time available for follow-up evaluation. One objective of the follow-up process is not just to evaluate the course or participants but to motivate them to try out new behavior. The fact that they would receive feedback on how successful their attempts have been seems to have made a difference in how hard they tried. This is why any course we conducted had a follow-up of some kind.

METHOD ONE - FREE ADVICE/EXPERT CRITIQUE

For skills courses such as Writing and Oral Presentations we used the first method, which is to convince the participants that part of the course is for them to send us a sample of their post-course writing or presenting.

This is easy for writing. They sent us a letter, report or other writing sample, we critiqued it and send it back with suggestions, congratulations or whatever feedback was appropriate.

Presentations are more difficult to critique. Sometimes alumni would videotape a presentation. We critiqued these videotapes and gave feedback over the phone. This was very effective. Also, when possible, we would sit in on a dry run and give some free advice about how to improve (with liberal amounts of reinforcement for improvements already made).

METHOD TWO - THIRD PARTY FEEDBACK/INSTRUMENTS

In our Leadership course (based on the Hersey & Blanchard model), there were pre-course leadership-style questionnaires. In class they received summary feedback from bosses, peers and subordinates. They used the data for deciding what behaviors they wished to change and how to go about it. We conducted post-course surveys eight to twelve weeks later. The same people who completed the pre-course questionnaires did so again. We then analyzed the results and sent personalized notes concerning the way they were now perceived by their followers. This is probably as motivating as having us periodically check with them (which would not be practical since, normally, we have other things to do and they work in other locations). Because they knew the post-test would be coming and their followers would be answering also, attention was once again focused on their behavior. It was a very powerful motivator since they knew their classmates who did well would be sure to share this good news with them and that if they give "school answers" instead of how they were actually behaving the others' questionnaires will reveal it. Those that got good news from the follow-up survey were not at all reticent in sharing it with other class participants. Those that didn't try to change or whose attempts were less effective got a good dose of social pressure to try or try harder. We loved it because they motivated themselves and we got the credit.

Resurveys of this nature are very powerful inducements to use skills/behavior that have been learned. They also provide an excellent reinforcement of alumni attempts to change. Combined with self-reports (see method three) this is a particularly useful follow-up process.

METHOD THREE - SELF REPORTS/INSTRUMENTS

Self-reports are notoriously lacking in validity. But, if that is all that is available, self-reports can serve as a useful follow-up motivator and reinforcer. Any of the self-report instruments available can be used for this purpose. But the vendors who provide compatible pre and post instruments make the job easier.

We asked our Leadership Course participants to complete a leadership style questionnaire. It is a "self" version of the questionnaire used in Method Two that is filled out by others with whom the participants interact. The post-class questionnaire is a different version. We were able to use the pre and post scores to evaluate knowledge gain and retention because the questionnaire was measuring the same information. The personalized coaching memos sent to each participant in Method Two include any changes in the "self" vs. "others" pattern of answers before and after class. Whatever their self-report indicated, we suggested they seek feedback from the people they interacted with.

Also, because the questionnaires are comparable with each other we were able to do a statistical analysis and show how the class as a whole had learned and retained the expected knowledge.

Self-reports can be used by themselves and provide some motivation to change. Also, they are easy to distribute, score and interpret. When you combine the self-report with third party feedback, you get a synergistic effect because of the pre-course disparity in self/other perceptions and the hopefully smaller post-course disparity.

METHOD FOUR - PROCESS FOLLOW-UP

In our Interviewing and Problem Solving courses we sent out a follow-up survey asking past participants to relate how they had applied what they learned in the class. We couldn't be there when they used the methods taught in the course, but from their descriptions we could tell if they were effectively using what they were taught. We sometimes wrote or called them and talked about the specifics in their follow-up survey responses. This extra attention was very well-received and resulted (we believe) in more motivation to change on their part. Also, after they attempted to use their new knowledge they had specific questions and concerns about application that never surfaced during the classroom instruction.

For the Problem Solving alumni we also had a follow-up case. Although the follow-up case was a voluntary activity, you would be surprised how many were willing to test themselves to be sure they learned the material well.

METHOD FIVE - CONTRACTS

In our Time Management class we asked all participants to contract with themselves to make a change that would increase time use effectiveness. The contract is a two-part carbonless form with four questions:

1. Having considered all my time problems I think the most significant time waster I face is:

2. This really is a time waster because:

3. This time waster occurs because I:

4. So, what I'm going to have to do is:

One copy of the contract was given to the trainer. A second copy was put in an envelope which they addressed to themselves. They kept the last copy. By committing to make a change, we hoped they would be more likely to make the change. Also, they knew we would send them their contract six weeks after the course.

Additionally, we sent them a follow-up survey with all the major Time Management areas from the course listed such as handling paper once, setting priorities, telephone use, etc. They were asked to indicate if they had attempted a change in each area, was it effective and had they continued to do it. Lastly, they were asked to estimate how much time they were saving per week because of the changes they made.

Typically, half of the class responded to the follow-up survey. The respondents usually tried many changes, most of which were effective. They report saving 2-3 hours per week on average. This is a tremendous return for a 5-hour class. Even if you discount the timesavings by 50%, there still is a savings of at least 400 hours for each 100 hours expended in a typical class of 20 participants.

We believe the contract, reminder and follow-up all serve to focus attention and reinforce those who are attempting changes. We had very specific results to show management, even if they would have done as well without the extra attention.

METHOD SIX - THIRD PARTY FEEDBACK - BOSS/OBSERVERS

One of the traditional methods of evaluating skills training has been to ask the participant's supervisor to report on changes in his/her subordinate's behavior after the class. We did that with our Writing and Oral Presentation classes. This is neither a unique nor complicated method of evaluation but it is often neglected.

In our experience, supervisors willingly report change or lack of change when questioned. It helped us know whether the skill lessons were transferred to the job. When positive change is noted, it reinforces the supervisor's good judgment in sending/or allowing subordinate attendance for training. When there were positive changes, we always suggested that the supervisor give feedback to the subordinate. The supervisor was asked to indicate what he or she had seen so that the changes were "celebrated" and reinforced. When there was no change or, worse yet, a negative change, we talked with the supervisor to analyze why this may have happened. We then worked with the supervisor to create a plan for coaching improved performance.

This method will also work with peer observers. We prepared a form for our Oral Presentation alumni to give to peers who observe their presentations. The form lists a number of presentation skills such as introductions, clarity of material presented, use of visual aids, etc. The observers are asked to indicate on a scale of ineffective to very effective (1-5) their rating of the person's presentations before the class and after the class. The observer feedback forms were sent to us. We summarized them and gave the feedback to the presenter. We probably received a lot of "friendly observations," but some had very specific comments (in a section provided). This is feedback that the presenters would not normally receive and can be very helpful. We also were able to get something of a reading of how well the training was transferred to the real world of the class participant.

DOES IT WORK?

One of these six methods will be appropriate for any management training that you may offer. The methods are effective because people have a greater likelihood of changing when they know someone else is watching, especially when they can measure it themselves. All of these methods are training evaluations, but more importantly they are participant evaluations. This is why they work. Or, so we believe. However, our belief is not the only reason we keep doing follow-up evaluations. Time and again employees will attend a course and after the follow-up process has shown positive results (and personal success) they send their subordinates to the course and they recommend the course to others. Follow-ups were one of the main reasons training was valued in the organization.

Post Script: This was written in the past tense since the company eventually was bought and broken up. During the time we followed up the training we were able to show management that there was a 400% ROI on training. We never would have known this without follow-up evaluations.

Thanks to Mike Smith for contributing this article. It is intended as information only and is not a substitute for legal or professional advice. Mike Smith is a Sr. HR Consultant with the Educational Testing Service at the time this article was written in November 2000.

http://www.shrm.org/hrresources/whitepapers_published/TOC.asp

-top of page-