Re: Training Feedback (a bit long)

[Steven Schwarzman <StevenS -at- Amdocs -dot- com>]

Probably the most commonly used model to measure success of training is by
Donald Kirkpatrick. 

That model has four levels. They go something like this:

Level one - reaction, or did the students like the course?
This is commonly measured by a questionnaire, aka a smile sheet, since
typically scores are falsely high. It doesn't tell you much.

Level two - learning, or did the students learn anything?
Unlike level one, this level measures what the students actually learn, as
measured by pre- and post-test. Many trainers are very pleased when the
scores improve from pre- to post-, but in most cases this is still not a
very good measurement. A low score can reflect a variety of causes - for
example, if they don't remember much of what you taught them, this might be
because your teaching was at fault. But it might also be because the content
was not what they needed. Or maybe the room was noisy. Conversely, a high
score still doesn't tell you that your course has the value you hope it
does. All it tells you is that the students can spit back at you what you
gave them.

Level three - behavior, or were the students able to apply what they learned
on the job?
Now we're getting to the interesting part. In contrast to levels one and
two, success on this level means that you are teaching the right stuff in
the right way. Of course, it's also a lot harder to measure. A quick and
dirty method would be to ask the students, but their answers may or may not
be entirely reliable. A more scientific approach would include empirical
observation, but most corporations don't have the time for this.

Level four - results, or did this training have a (hopefully!) positive
impact on the bottom line?
Clearly the hardest to measure, because it is the question you or your boss
most want to have answered! Like in level three, possibly the closest you
can get to measuring this without devoting inordinate amounts of time is to
ask the students, their bosses, or other appropriate people in the
organization. Other ways might include measurements of productivity, etc.
But these also have to control for other factors - for example, perhaps the
students began working on faster PCs during the period being measured, or
whatever.

So much for the theory. As for practical advice, like what to put on your
questionnaires: include questions on every aspect of your class that you can
think of. Don't ask "was the instructor good?". Ask "did the instructor know
the material?", "did the instructor answer your questions well?", etc. Write
detailed questions about the instructor, the materials, the venue, the
timing, the coffee, etc. - only in these details will you get any
information that you can actually use to improve your course. If you stick
to a one-pager with generalities, you'll get "good" (remember the smile
sheets) simply because it's easier for the student to finish quicker that
way and leave the room. Our questionnaire is currently about three pages,
and includes questions that try to measure at least the students' OPINIONS
about level three and four success. After every course, we tabulate the
scores for each question to see if there are any significant deviations from
the usual good-to-very-good rating.

Finally, although you didn't ask, keep your training - whatever the subject
- as hands-on as possible. This will help you move toward level-four success
- real impact. If you train theoretically, you may get good level one or two
scores, but you won't accomplish much. If you have to choose between more
hands-on exercises and covering all the material, go for the exercises. It
is better to successfully teach the most important subjects than to fail by
trying to cover them all. You can put the extra material in a user guide or
help for future reference.

Hope this helps.