Everything you need to know about training evaluation.
Training evaluation can be defined as any attempt to obtain information (feedback) on the effects of training program and to assess the value of training in the light of that information for improving further training.
Evaluation of training can be viewed as a method of measuring change in knowledge, skills, attitudes, job performance, costs and the quality of the training facilities.
A major reason to evaluate training programs is to determine whether the training programs are accomplishing their specific training objectives.
The process of training evaluation has been defined by A. C. Hamblin as “any attempt to obtain information on the effects of training performance and to assess the value of training in the light of that information”.
Thus, evaluation of training effectiveness refers to the process of obtaining information on the effects of a training programme and assessing the value of training in the light of the information so obtained.
Learn about:-
1. Introduction and Meaning of Training Evaluation 2. Definition of Training Evaluation 3. Criteria 4. Purposes and Principles 5. Process
6. Evaluation of Training Manager 7. Methods 8. Kirkpatrick’s Model 9. Effectiveness of Training Programme.
Training Evaluation: Introduction, Meaning, Levels, Purpose, Principles, Process and Methods
Contents:
- Introduction to Training Evaluation
- Meaning of Training Evaluation
- Levels of Training Evaluation
- Purposes and Principles of Training Evaluation
- Process of Training Evaluation
- Evaluation of Training Manager
- Methods of Training Evaluation
- Kirkpatrick’s Model of Training Evaluation
- Evaluating the Effectiveness of Training Programme
Training Evaluation – Introduction
Training evaluation has a far higher profile now than it did in the past. As a profession, training has long promised- ‘Give us the resources and we’ll transform the business.’ Line management is now replying- ‘Prove it!’ There is a requirement to prove the connection between an investment in training and an improvement in organisational performance. While this is understandable, it creates some difficulties for trainers.
The fact is that the benefits of sustained long-term investment in training are usually impossible to calculate accurately. An organisation that has sanctioned a major increase in training expenditure will also be doing other things differently- there will be new managers, new products, and new markets, and so on.
However, it is not acceptable to use this as a rationale to justify lack of accountability, and trainers must be able to make some estimation of the impact of their efforts or lose credibility.
There is, therefore, a strong case for attempting to evaluate training, particularly in view of the very large sums of money that are spent on it. However, there are a number of problems associated with the evaluation process which must be considered.
The first difficulty is that, in an ideal world, it would be necessary to measure the exact knowledge and skill of each trainee before the start of the training. Without this information it is impossible to assess what has been learned by the end.
What someone is capable of doing at the end of the training may primarily reflect what they could do before the training. To separate out the new learning may necessitate a pre-test, which is practicable in some learning situations but becomes much more difficult in other situations.
For example, if we were to introduce the pre-testing of senior managers before a course on leadership, we could anticipate some resentment which could actually inhibit learning. With subjects such as assertiveness, someone’s ability to display assertive behaviour could be greatly reduced by the anxiety generated in the assessment process.
Pre-testing in many situations may also inhibit the process of establishing rapport with the course members, and can result in the learning experience becoming a ‘What do I have to do to get through it?’ ordeal. Sensitivity must be applied to any assessment process.
Another difficulty is that an on-going review tends to result in changes to the detail of the programme before it can be thoroughly evaluated. It is not sensible to say to line managers ‘I know it’s not working, but I want to prove that systematically before changing it.’ Sometimes a rigorous evaluation methodology must be sacrificed for the sake of expediency.
A third difficulty is the sheer workload that thorough evaluation can require. Although evaluation is important, is it more important than delivery or design? Many line managers can be convinced of the importance of evaluation, but most would not want to see it taking up more than a small proportion of the trainer’s time. They would rather see you training than evaluating.
Before approaching an evaluation project, we must ask ourselves-
1. Why is the training evaluation required?
There are various reasons for evaluating training:
i. The evaluation enables the effectiveness of an investment in training to be appraised which can help to justify expenditure on future programmes.
ii. It allows the effectiveness of differing approaches to be compared.
iii. It provides feedback for the trainers about their performance and methods.
iv. It enables improvements to be made, either on the next occasion, or if the evaluation is ongoing, as the training proceeds.
v. Recording learning achievements can be motivational for learners.
vi. The evaluation indicates to what extent the objectives have been met and therefore whether any further training needs remain.
2. What should be evaluated, and when?
A number of different models have been developed by various writers, some of which are described below to answer this question. Knowledge of at least one of these models or frameworks is necessary to help to get to grips with the evaluation process.
Training Evaluation – Meaning
Training evaluation can be defined as any attempt to obtain information (feedback) on the effects of training program and to assess the value of training in the light of that information for improving further training.
Evaluation of training can be viewed as a method of measuring change in knowledge, skills, attitudes, job performance, costs and the quality of the training facilities. A major reason to evaluate training programs is to determine whether the training programs are accomplishing their specific training objectives.
A training program that does not change employees’ knowledge, skills or attitudes in the desired direction should be modified or replaced. A second reason for training evaluation is to assure that any change in the trainees’ capabilities is due to the training program and not to other conditions. In order to determine that a training program is responsible for changes in trainees, it is necessary to compare the trainees’ performance before and after the program with a control group.
Evaluation of a training programme becomes necessary to find out how far the training programme has been able to achieve its aims and objectives. Such an evaluation provides useful information about the effectiveness of training and the design of future training programme.
Such an evaluation also guides the management to update or modify its future training programmes. It also enables the management to collect useful data about the success or otherwise of the training programme and on the basis of such data, the management can judge relevance of training and its integration with other functions of personnel management.
Training effectiveness refers to the degree to which trainees are able to learn and apply the knowledge and skills which they have acquired in the training programme. The effectiveness of training depends upon such factors as the attitudes, interests, ability, willingness, values and expectations of the trainees and the training environment.
The training programme will be more effective if the trainees are eager to learn, if they are involved in their jobs and if they have career strategies. Further, contents of training programme, teaching methods, trainers’ ability to teach well and their ability to motivate also determine the effectiveness of training.
Training Evaluation – Levels: Pre-Training Evaluation, Intermediate Training Evaluation and Post-Training Evaluation
Training segment has to evolve criteria for evaluating the impact of training on employees. Generally four different criteria are used to evaluate training programme namely reaction of trainees, knowledge acquired, behaviour modification and other job performance parameters like reduced accidents, increased productivity, lowered absenteeism leaping sales etc.
1. Pre-Training Evaluation:
In this stage, an evaluation is made in the beginning of the training programme in order to understand the expectations of the trainees from the training programmes and the extent to which they have understood its objectives. This step enables the training segment to modify the training curricula in such a way that the objectives of the training programme are aligned to those of the trainees.
2. Intermediate Training Evaluation:
Training and development segment wants to ensure that training is progressing as expected. Mid-course corrections can be made in the event of deviation from the envisaged objectives. For example, if trainees perceive that a training programme is aimed at building communication skill is more theory-oriented, rather than practice-oriented, the feedback may be useful to modify the instruction method. Thus, it serves as a verifying tool.
3. Post-Training Evaluation:
The criteria used for assessing the impact of training programme include Reaction, Learning, Behaviour and Results (RLBR).
a. Reactions:
This measures the degree of satisfaction of trainees with the training programme, namely subject matter and content of training programme, the environment, methods of training etc. The outcome of evaluation of reaction may be useful in further strengthening the areas the participants find it more useful and in modifying the areas they find it not useful. Negative reactions may dampen the spirit of participation in future training programmes. However, positive reactions may not provide complete information about the effectiveness of the programme.
b. Learning:
It measures the degree to which trainees have acquired new knowledge, skill or competencies. The trainer has to measure the knowledge and skill level of trainees in the beginning of the programme. It is supposed to be the baseline or standard. Again the level of knowledge and skills obtained at the end of training is measured and compared against the standard. Thus pre and post training comparison helps to assess the improvement level.
c. Behaviour:
Similarly, a comparison of pre and post training behaviour may reveal the impact of training on behaviour modification. Yet, unfortunately, much of what is learnt during training cannot be used on the job owing to lack of resources or conducive environment. In such a case, one cannot say that the training is ineffective. It follows that when training environment is similar to actual work environment, such a climate facilitates transfer and application of learning.
d. Results:
Generally, it is difficult to measure precisely the impact of training on business performance which depends on several other factors like economic climate, marketing, size of investment, etc. However, certain measures like productivity, sales volume and profit, etc., may be compared before and after the training episode. Any improvement may be partially attributed to the training imparted. Besides, return on investment, cost benefit analysis and bench-marking are other methods to assess the value of training.
Training Evaluation – Purposes and Principle
Certain purposes guide and dictate the need for evaluation; the purposes behind training evaluation are wide enough. Training evaluation will have at least one purpose as its primary focus. If the training cannot fulfil achieving the planned business need, then you are required to identify the root cause and take remedial measures.
The purposes of training evaluation are as follows:
1. To justify the role of training, considering budget availability and cutback situations
2. To improve the quality of training for employee development, training delivery, trainer deployment, duration, methodology, etc.
3. To assess the effectiveness of the overall programme, quality, and competency of the trainer
4. To justify the course through cost-benefit analysis and ROI approach
The evaluation data, once collected, takes many forms, and is highly valuable.
It can also be used to do the following:
1. Provide feedback on whether the training or development activity is effective in achieving its aims
2. Indicate the extent to which trainees apply what they have learned back in the workplace (transfer of training), an issue which many organizations find they have problems with
3. Provide information on how to increase the effectiveness of current or later development activities
4. Demonstrate the overall value and worth of development activities.
The purposes of evaluation further extends to gauging the effectiveness, effects of training on shaping attitudes, improving performance, reducing rejections, lowering machine downtime, enhancing job quality, enhancing the market share, penetrating new markets, increasing sales, improving quality of work life, promoting interpersonal communication, etc.
The preceding factors speak about the complexity of any effort to evaluate training. These factors further emphasize the importance of being clear about the purpose and the process of evaluation.
In fact, effective evaluation must be carefully planned while designing the training. Obviously then, the evaluation plan should precede training and not follow it. Meticulously planned and well-conducted evaluation provides useful information to the institute, the trainer, participants, and sponsoring organizations.
The output of training evaluation will serve:
1. To illustrate the real worth of a training
2. To pinpoint where improvement is required in forthcoming training programmes
3. To assess effectiveness of the overall course, trainer, and the training methods
4. To carry out cost-benefit analysis to justify the amount spent; to prove that the benefits outweigh the cost
5. To formulate a basis for making rational decisions about future training plans
6. To justify the role of training for budget purposes and in cutback situations of budget crunch.
Summarily, you are required to improve the quality of the training, concentrating on the trainers’ competency, training design, content of the course, participant profile, expected behavioural outcomes, methods used, length of training, achievement of the training objectives by means of improving the training delivery as a whole. Knowing the purposes of training evaluation, you will now be acquainted with the process through evaluation.
Principles of Training Evaluation:
1. Evaluation specialist must be clear of the training program and also about the goals and purposes of evaluation.
2. Evaluation should be continuous.
3. Evaluation must be specific.
4. Evaluation must provide the means and focus for trainers to be able to appraise themselves, their practices, and their products.
5. Evaluation must be based on objective methods and standards.
6. Realistic target dates must be set for each phase of the evaluation process.
Training Evaluation – 3 Steps in the Process of Evaluating Training: Before Training, During Training and After Training
The process of training evaluation has been defined by A. C. Hamblin as “any attempt to obtain information on the effects of training performance and to assess the value of training in the light of that information”. Thus, evaluation of training effectiveness refers to the process of obtaining information on the effects of a training programme and assessing the value of training in the light of the information so obtained.
It involves controlling and correcting the training programme. The basis and mode of evaluation are determined when the training programme is designed. According to Hamblin A. C., there are five levels at which evaluation of training can take place.
i. Before Training:
Generally the HR manager or the employee’s supervisor appraises the employee’s skills and knowledge before the training programme. Employee is asked to give his/her opinions on the methods of the training used and whether those methods confirm to his/her preferences and learning style.
ii. During Training:
This is the step which instruction is started. This step usually consists of short tests at regular intervals.
iii. After Training:
This is the step when employee’s skills and knowledge are assessed again to measure the effectiveness of the training. This phase is designed to determine whether training has had the desired effect at individual department and organizational levels. There are various evaluation techniques for this phase.
Evaluation helps in controlling and correcting the training programme. Hamblin(1974) suggested five levels at which evaluation of training can take place.
These are:
1. Reactions- This measures participants’ reaction to the training at the time of training. Whether they like or dislike the training programme? Trainee’s reaction to the overall usefulness of the training includes coverage of the topics, the method of presentation, the techniques used to clarify things and effectiveness of the programme.
2. Learning-This measures participants’ learning of the content of the training. The extent to which the trainees have assimilated the knowledge offered and skills practiced in the training programme. Does the participant score higher on tests after the training or development than before?
3. Job Behaviour-This indicates participants’ use of their new skills and knowledge back on the job. This includes a comparison of ratings; a participant receives before training and after training.
4. Organization-This measures participants’ use of training, learning and change in the job behaviour of the department/organization in the form of increased productivity, quality, morale, sales turnover and the like.
5. Ultimate Value-This measures contributions of training programme to company goals like survival, growth, profitability etc. and to the individual goals like development of personality and social goals like maximizing social benefits.
Training Evaluation – Evaluation of Training Manager
Of the many HRD mechanisms or instruments, training is by far the most vital mechanism. Training must be planned and evaluated using any standard model or combination of them, depending on the organizational needs and practices. The evaluation is required at many stages—before the implementation of the training, during conduction of the training or management development programme, immediately after completion of the programme, and even after a few months.
Evaluation literally means the assessment of value or worth. In simple terms, it is the act of judging whether or not the activity to be evaluated is worthwhile in terms of the set criteria.
Indiana University’s centre for research on learning and technology (CRLT) recently used seven principles to evaluate the four online courses being conducted in a professional school at a large mid- western university.
These were principle 1 —good practice encourages student-faculty contact, principle 2—good practice encourages cooperation among students, principle 3—good practice encourages active learning, principle 4—good practice gives prompt feedback, principle 5—good practice emphasizes time on task, principle 6—good practice communicates high expectations, and principle 7—good practice respects diverse talents and ways of learning.
Lynton and Pareek (2007) have emphasized five sets of objectives and three main questions for such evaluation.
The five sets of objectives include:
1. Evaluating the training programme (to improve the programme as a whole, to improve major components of the training programme, and to increase the effectiveness of specific modules or sessions),
2. Evaluating the training process (to improve the training climate, to improve the training methodologies, and to improve the trainers team),
3. Evaluating the training execution/conduction (to improve the training facilities and contextual factors),
4. Evaluating training outcomes/impacts (to increase learning by individuals participants, to increase use of learning in work performance, to contribute data to organizational effectiveness, and to help the organizational change), and
5. Evaluating programme factors (to maximize effectiveness, to ensure post- training support, and to identify helping and hindering factors).
In order to make a training programme effective, the training objectives must find answers to three questions—for whom, when, and how.
In a usual course of action, an organization first identifies the training needs for its employees. Then, it designs the training programme, considers the pedagogy or andragogy, and thereafter, conducts the training programme. All the steps need evaluation for effectiveness of the programme.
The evaluation is required from the training design stage, before conducting the training, during conduction of the training or management development programme, immediately after completion of the programme, and even after a few months to a year. Evaluation is an essential part of planning the training.
It is important to not confuse between evaluation and assessment. Assessment is about the progress and achievements of the individual learners, whereas evaluation is about the learning programme as a whole. Training evaluation constitutes the most systematic way of determining the strengths, weaknesses, and effects of training courses. Few trainers make use of this process on a continuing basis.
Managers of training need to understand the practical benefits of training evaluation so that they will be motivated to support its use.
Allahabad Bank evaluates its training sessions based on the contents and coverage of the programme, methodology used, reading material provided for, presentation skills, utility of the contents, topics liked by the participants, infrastructure of the institute, etc.
Training Evaluation – 2 Important Methods for Evaluating Training: ROI of Training and Training Hours per Employee (With Formula)
The various methods of training evaluation are observation, questionnaires, interview, cost benefit analysis, self diaries and self recording of specific incidents.
Method # 1. ROI of Training:
About $5.6 billion to $16.8 billion is wasted annually on ineffective training Programmes. “American industry is spending billions and billions on training programmes and doing no evaluation of their effectiveness. You have to measure it.”- Cary Cherniss, Rutgers University
Employee training is responsible for productivity improvements, greater workforce flexibility, savings on material and capital costs, improved quality of the final product or service, and a more motivated workforce. To know these it is necessary to calculate the Rate of Return on a firm’s Investment (ROI) of training.
Apparently there is no other workplace issue on which so much money is spent with as little accountability as training.
Alternative Training Evaluation ROI Methods:
Alternatives to ROI are increasingly being used to measure the success of training programs.
Some of these methods include the following:
1. 360-Degree Appraisal Feedback Process – This performance evaluation system uses input from all employee levels (appraisers, supervisors, and peers) to assess performance.
2. Performance-Learning-Satisfaction Evaluation – This method uses both conceptual components and data processing to assess business results at an organizational level and financial results in terms of monetary ratios. Its creators claim the model has a 2:1 return on investment.
3. The Balanced Scorecard – This method tracks the key elements of a company’s strategy using both financial and operational measures. The balanced scorecard considers four primary questions- How does the company look to the shareholders? (Financial perspective); How do the customers see the company? (Customer perspective); at what must the company excel? (Internal perspective); Can the company continue to improve and create value? (Learning perspective).
Research also highlights other methods to creatively measure training’s ROI.
Such methods include the following:
1. The Success Case Method—Demonstrates business value through credible stories that show economic impact on the company. This method relies on short, behaviour-based questionnaires which trainees complete post-training. The survey is designed to test some fundamentals about what the learning was supposed to deliver. Low success cases help the training department determine whether it is the training program or an unrelated issue that may have caused the low impact.
2. Show the Payback—Measures the payback period of the training initiative to show how long it will take to recover the investment the organization initially made. Two major drawbacks are that such calculations ignore the opportunity cost and the time value of money.
3. Evaluate the Consequences of Not Training—Demonstrates how not training the workforce will impact the organization in terms of competition, productivity, product marketing, and equal opportunity violations. The evaluator must identify a potential loss or problem and predict its financial or business impact while isolating how the lack of performance will have an impact, as well as the non-performance factors and their costs. Such costs are then compared to the benefits of the training program in terms of estimated costs. This method can be somewhat subjective and is not appropriate for all training programs.
4. 80/20 Rule—Estimates the soft benefits like customer satisfaction with 80 per cent accuracy using only 20 per cent of the effort. Testing and surveys can measure soft benefits like morale and increased employee satisfaction as a result of the training program.
5. Merck and Company uses a unique formula, detailed below, to measure the impact of specific training programs. With this model, Merck determined that its average ROI for training programs is 84 per cent and terminated 53 programs that were not producing high enough returns.
GAIN = D x SD$ x JSI x N
Where,
D = shift in performance by average individual undergoing training expressed in standard deviations from pre-training average,
SD$ = the value in dollars of one standard deviation of performance shift,
JSI = percentage of job skills affected by training.
N = number of participants who underwent training.
Method # 2. Training Hours per Employee (THPE):
This is a measure of the average number of hours spent by an organization’s employees on training activities.
The number of training hours provides a good gauge of the company’s overall training efforts. It is an indirect measure for assessing the value the organization places on its workers.
Training Evaluation – Kirkpatrick’s Model of Training Evaluation
Level 1 – Reaction:
Questionnaires, interviews, group discussion, or asking trainees to write a report can be used. Care must be taken with all of these methods. Very often participants have enjoyed a course, even if they learned very little. Factors such as the quality of the lunch provided, or the comfort of the chairs, may influence the assessment of the training given. The other participants may have spoilt a basically sound course, or conversely saved a basically poor course.
Trainees are not always in a position to know immediately whether what they have learned will be useful and it may be best to wait some considerable time before asking for an opinion. Sometimes a trainee may have felt unfairly criticised during a course, and so may ‘rubbish’ it in retaliation. The more training a person receives, the more critical he or she is likely to become. Standards and expectations rise with experience.
Using more than one technique can be helpful to gain a broader picture. Also look out for cues such as an increase or decrease in demand for the training (where there is a choice), or if the line managers start asking for one particular trainer in preference to another.
Level 2 – Learning:
Tests, examinations, workplace-based assessments of competence, projects, or attitude questionnaires are the key techniques here. Some learning situations are easy to test for (e.g typing ability), whereas others necessarily involve a good deal of subjectivity (e.g counselling skills).
Yet other learning is so long-term in its nature that direct methods are frankly not appropriate. For example, if a newly appointed supervisor attends a course, then an end test or examination can only tell us if he or she has learned certain terms, concepts or models. It cannot tell us if he or she will become a good supervisor by applying that learning in the work situation.
The processes used at level 2 are often termed validation.
Level 3 – Behaviour:
This level requires assessment of improved performance on the job. This is easiest in jobs where before-and-after measures can easily be made (e.g the speed at which an insurance proposal form can be processed).
It becomes more difficult to evaluate performance in jobs which are less prescribed and where measurement is imprecise (e.g training design). There may be a time-lag between training and the appearance of indicators of performance improvement.
For instance, upon returning to work after attending a course on leadership, a manager may immediately practise what he or she has learned – but the results of this take two or three months to become apparent.
During that time other factors in the situation may have changed – there may have been some new staff recruited, or some redundancies have affected morale. If we were to instigate a long-term assessment process, we would also find it difficult to separate out the influence of day-to-day experience from the influence of the formal training course, it is often impossible to isolate the precise influence of the training. Often the trainer has to resort to indirect performance assessment measures to gauge the influence of the training.
Level 4 – Results:
Because departmental and organisational results depend upon many people and it is difficult to attribute improvements to the efforts of specific individuals, evaluation at this level often has to be conducted in a more general way.
Does the overall training programme result in greater efficiency, profitability, or whatever? If we were to try to look at the impact of a large training programme on a part of a large organisation, we can take an experimental approach.
Ideally, we take two identical units. One is given lots of training, the other is given none. Two years later, the difference in performance is apparent!
Obviously such an approach is not one which can be easily advocated. If we really believe that the training is likely to be of value, it is unfair, perhaps even unethical, to withhold it from one of the units in order to conduct an experiment.
However, it is sometimes possible to obtain historical information which shows a correlation between spending (or some other measure) on training and organisational performance. Perhaps two similar units within the same organisation can be compared and the relationship between past training activity and other measures can be assessed (e.g. accident rate, machine downtime, customer complaints).
Training Evaluation – How to Evaluate the Effectiveness of a Training Programme?
There are several methods of evaluating the effectiveness of a training programme but the following methods are important:
1. Assessment of trainers’ comments and reactions to the training programme after the training is over.
2. Observation of trainees during the training programme.
3. Comparing on-the-job performance of the trainees before and after training.
4. Collection of opinions and judgements of trainers, superiors and peers.
5. Giving oral and written tests to trainees to find out how far they have learnt through the training programme.
6. Cost-benefit of the training programme.
7. Measurements of levels in employees’ absenteeism, turnover, productivity, wastage or scrap of materials, accidents, breakage of machinery during pre-training period and post-training period.
8. Evaluation of trainees’ skill level before and after training.
9. Collection of opinions of the trainees’ subordinates regarding their job performance and behaviour.
10. Collection of information through evaluation forms duly filled up by the trainees.
11. Knowing trainees’ expectations before training and collecting their views regarding the attainment of the expectations after training.
Management can make training programme effective, if following guidelines are strictly followed:
1. Training objective to be specific to performance standards of trainees to achieve organisational objectives
2. Proper screening of personal needs in comparison to operational needs will give an added advantage to establish actual needs of individual employees.
3. Efforts should be made to establish where the trainee has the required intelligence, maturity and also the motivation to successfully complete the training programme. If this is not noticed among trainees the training programme may be postponed or cancelled till adequate improvements are noticed.
4. The trained should be encouraged to see the training by making him aware of the personal benefits he can avail through better performance.
5. Training programme to be planned so that it is related to the trainees’ previous experience and background. This background may be used as a base for new advancement in professional career.
6. Efforts to be taken to create friendly atmosphere that is conducive to good learning environment. Any deviation in the process of environment to be avoided well in advance. Permission to be obtained from top level management before commencing the training at lower levels.
7. It is to be understood that all levels of trainees do not progress at the same speed; hence, flexibility should be shown in judging the rate of progress in the training programme.
8. The personal involvement of trainees, as far as possible be encouraged in the training programme. He should be given opportunity to participate to have adequate practical knowledge in the newly needed behavioural norms.
9. As the trainee’s acquisition knowledge, skills, attributes and utilise them while executing the job. At the same time the trainee should be motivated for better performance and also be given necessary incentives.
10. Trainees to be given personal guidance as and when needed so as to help obviate learning obstacles.