MW Canada is an innovative and leading manufacturer of textile window covering textiles and speciality materials located in Cambridge, Ontario. The company's 65 employees manufacture window coverings, filtration composite, and energy-related fabrics that are sold to manufacturers and wholesalers. The company uses advanced technologies for specialty materials; thus, ongoing training is critical.

In an effort to adapt to increasingly competitive markets, MW Canada implemented a new learning initiative in 2013 called the Technical Skills Certificate Program (TSCP). The program was designed to improve communication and discussion of technical issues and improve problem solving of its production staff. The two main business objectives were to improve product quality and reduce equipment downtime.

The 90 hours of training was delivered in two-hour sessions twice a week over six months. The program consisted of a blended delivery method that included online modules, in-plant instructor-led methods for hands-on training, and on-the-job coaching with MW Canada production managers.

Following the training, production managers observed an increased level of independent thinking in the workplace as a result of the skills training program. For example, loom mechanics now start jobs "without requiring supervisors to coax or push them." In the morning, upon reviewing the tasks of the day, "employees pick their jobs and independently start them." And addressing complex problems has become a great deal easier "since staff feel they have more knowledge and more confidence."

In addition, approximately three-quarters of those who were trained applied their learning on the job, improved their performance, and perceived the training as a worthwhile investment for MW Canada. And just more than half of the participants reported that the training improved organizational outcomes such as quality and productivity.

In terms of the two main business objectives, improving product quality and a reduction in downtime, the total annual savings from improvement in these two areas was $20,400. With a total cost of $17,500, the return on investment of the TSCP program was .17, or 17 percent. In other words, every dollar spent on training was returned plus an additional $0.17 as a result of reducing equipment downtime. However, the return to the organization will be much higher, since the benefits will continue to accrue in subsequent years.1

Sources: Gillis, L., & Bailey, A. (2014, January).Workplace literacy and essential skills research: MW Canada Ltd. Technical Skills Certificate Program Impact Analysis. Canadian Manufacturing Network, Div. of Excellence in Manufacturing Consortium; Giroux, J. P. (2014, October 3). MW CanadaEssential Technical and Manufacturing Skills Certificate. Canadian Manufacturing Network, www.emccanada.org;Executive summary, MW Canada: Technical Skills Certificate Program. Canadian Manufacturing Network,Workplace Literacy and Essential Skills Research Project (WLESR), www.emccanada.org

// Introduction

Organizations have become increasingly concerned about the costs and benefits of their training and development programs, and the value added to the bottom line. As you can see from MW Canada's Technical Skills Certificate Program (TSCP), it is possible to calculate the benefits of training programs in monetary terms and the return on training investments. For MW Canada, the return on investment of the TSCP was 17 percent.

Several years ago, Accenture, a global management consulting firm, demonstrated that the net benefit of its training and development programs was $1.27 billion, and that for every dollar invested in training there was a return of $3.53, or an ROI of 353 percent.2Thus, training can have a substantial financial impact on an organization's bottom line by maximizing the return on training.

Information on the costs and benefits of training programs is an important part of training evaluation. In fact, some experts consider the calculation of return on investment (ROI) to be Level 5 in Kirkpatrick's model of training evaluation (see Chapter 10).3

In this chapter, you will learn how to calculate the costs, benefits, net benefit, benefit-cost ratio, return on investment (ROI), and utility of training and development programs. But first, let's discuss training and the bottom line.

// Training and the Bottom Line

In Chapter 10, we described the process of training evaluation. This usually involves measuring trainees' reactions, learning, and behaviour, and organization results. The intent is to show some improvement in employees' knowledge, on-the-job behaviour, and organizational outcomes. Typically, one hopes to see an improvement in employees' learning and on-the-job behaviour and a positive effect on organizational outcomes such as sales or customer satisfaction.

But what about the cost of a training program? What if a training program is very expensive? That is, what if the cost of training is greater than the benefits? Would improvements in employee behaviour and organizational outcomes be worthwhile? Would the training program be worth the cost?

Without information on the cost of training and the monetary value of training benefits, one cannot adequately answer these questions. Clearly, the effectiveness of a training program also depends on its costs and benefits. Managers might be pleased to know that a training program has improved customer satisfaction, but they will be even more impressed by the financial value of an improvement in customer satisfaction.

Costing is a complex and time-consuming process that many training specialists prefer to avoid. Some managers are skeptical about the theoretical underpinnings of costing, while others suggest, rightly, that in business not everything is quantifiable. Indeed, many managers suggest that some attitudes and behaviourssuch as job satisfaction and communication techniquesmake people feel good about themselves and the company they work for, and you just cannot put a dollar value on them.

However, there is increasing pressure for human resource and training professionals to demonstrate the financial value of their programs. Organizations increasingly want to know the return on their training investments, and in a time of declining budgets, the training function faces increased pressure to justify the costs of training programs.4In fact, a survey of CEOs found that 96 percent of them want to see learning and development connected to business impact data. And while 74 percent want to see ROI data, it happens in only 4 percent of cases.5

Therefore, trainers and HR professionals must learn how to calculate and indeed demonstrate the financial benefits of training and development investments. This shows the value of training programs to management and the organization and it also justifies the training function's share of the budget. It also improves its status and influence within the organization.6

In addition, other members of an organization are more likely to see training and development as an investment rather than a cost, and training budgets are less likely to be slashed during economic downturns when the financial benefits of training and development programs can be clearly demonstrated.

You might recall from Chapter 1 that many organizations in Canada view training and development as an operating expense or cost that should be minimized rather than as an investment. In fact, most executives view training and development as a cost rather than as an investment.7

Calculating the costs and benefits of training programs is necessary for viewing training as an investment. Financial information about the benefits of training programs places human resource and training professionals on an equal footing with other functional areas in an organization.

The starting point for demonstrating the financial benefits of training and development programs is to identify the desired end result of training. To learn more, see The Trainer's Notebook 11.1, "Business Impact-Driven Learning."

THE TRAINER'S NOTEBOOK 11.1

Business Impact-Driven Learning

Effectively managing learning programs involves a bottomline approach that starts with clearly identifying the end goal of trainingspecifically, how well employees are applying their knowledge to the job and achieving target job outcomes as a result. This is called business impact-driven learning. Business impact-driven learning involves four steps:

• Identify the job performance outcomes or results you expect from your employees. Start with the end result you want to achieve: business outcomes and performance objectives.
• Define how they will achieve those resultskey behaviours employees must exhibit continually to affect the business. Define behaviours associated with performance objectives.
• Design a learning program to teach employees the knowledge they need to perform the desired behaviours.
• Collect the right data to actively track behaviours and measure outcomes; identify deficits that are affecting performance; and design and adjust training in real time to ensure that employees consistently have the knowledge needed to perform to their potential.

// Training and the Bottom Line in Canadian Organizations

Although most training programs in Canada are evaluated at Level 1 (trainee reactions), only 20 percent of organizations conduct Level 5 (cost-benefit analysis) evaluations and only 19 percent of courses are evaluated at this level.8However, more organizations, such as MW Canada, are beginning to conduct financial evaluations of their training and development programs.

For example, several years ago supervisors at Firestone Canada attended a problem-solving workshop after a lack of skills was identified as the source of an internal problem. After the training the supervisors used their new skills to solve the problem. An assessment of the program's impact on the supervisors indicated that supervisors acquired new problem-solving skills, strengthened their analytical thinking, participated in more dynamic teamwork, and had better quality control. As for the bottom line, the company realized a cost savings of $10,000 per month.9

In order to calculate the effect of training and development on the bottom line, you must calculate the costs and benefits of training and development. In the next section, we describe how to calculate training and development costs; in subsequent sections we describe how to calculate the benefits of training and development.

// Costing Training Programs

Costingis the process used to identify all the expenditures used in training. This is an important procedure in both the design and evaluation of a training program.

The cost of MW Canada's TSCP was $17,500. But where exactly do the costs come from? The calculation of the cost of a training program usually involves the assignment of various costs to a number of meaningful cost categories. Over the years, a number of approaches have been developed. One approach categorizes the costs of training according to the stages of the training process. For example, one might calculate the cost of needs analysis, training design, delivery, and evaluation. These costs are usually listed on a costing worksheet. One can then calculate and compare the cost of each stage as well as the total cost of a training program.

Another approach to costing training programs is to categorize the costs according to the nature or kind of cost. An example of this approach uses the following five cost categories: direct costs, indirect costs, development costs, overhead costs, and trainee compensation costs.10

Direct Costs

Direct costsare costs that are linked directly to a particular training program. This includes the trainers' salary and benefits, equipment rental, course materials, instructional aids, food and refreshments, and the cost of travel to and from the training site. These costs are so directly linked to a particular training program that they would not be incurred if the training program were cancelled.

Indirect Costs

Indirect costsare costs that are not part of a particular training program per se but are expenses required to support training activities. Indirect costs include clerical and administrative support, trainer preparation and planning, training materials that have already been sent to trainees, and the cost of marketing training programs. These costs would still be incurred even if a training program were cancelled. In other words, unlike the direct costs, these costs cannot be recovered.

Developmental Costs

Developmental costsare incurred in the development of a training program. This would include the cost of doing a needs analysis, the cost of developing instructional media such as video, the design of training materials, and the cost of evaluating a training program.

Overhead Costs

Overhead costsare incurred by the training department but not associated with any particular training program. Such costs are required for the general operation of the training function, such as the cost of maintaining training facilities (e.g., heat and lighting) and equipment and the salaries of clerical and administrative support staff. A portion of these costs must be allocated to each training program.

Trainee Compensation Costs

Trainee compensation refers to the cost of the salaries and benefits paid to trainees while they are attending a training program. This might also include the cost of replacing employees while they are in training. The logic behind this cost is simply that employees must be paid while they are not working and this is a cost of the training program.

// Training Costs at the Wood Panel Plant

Table 11.1 presents a training cost analysis using the five cost categories described above. The example is from a company that produces wood panels. The company had three problems that it wanted solved. First, it wanted to improve the quality of wood panels because they were experiencing a 2 percent rejection rate each day due to poor quality. Second, they wanted to lower the number of preventable accidents, which was higher than the industry average. Third, they wanted to improve the housekeeping of the production area, which was considered poor and a cause of some of the preventable accidents. Visual inspections that used a 20-item checklist indicated an average of 10 problems in housekeeping each week.11

The solution was to train supervisors in performance-management and interpersonal skills. Forty-eight supervisors, seven shift superintendents, and a plant manager attended a three-day behaviour-modelling skill-building training program. The objectives of the program were to teach the supervisors how to discuss quality problems and poor work habits with employees, to recognize improvements in employee performance, to teach employees on the job, and to recognize employees for above-average performance.

The cost of the training program was calculated for each of the five cost categories. As shown in Table 11.1, the total cost of the training program was $32,564, or $582 per trainee. This was based on total direct costs of $6,507; indirect costs of $1,161; development costs of $6,756; overhead costs of $1,444; and compensation costs of $16,696.

It is important to recognize that the costing process presented in Table 11.1 is only an example. It represents one approach for categorizing training costs and might need to be modified to suit an organization's unique circumstances. The idea is to identify the main costs of a training program and not to worry too much about the labels assigned to them. The trainer should be most concerned about how to design a costing approach that has credibility within the organization and that will be accepted by management.

Once the costs of a training program have been calculated, they can be used for at least two purposes. First, they can be used to get budget for a training program and to compare and contrast the costs of different programs. This is important when making decisions about whether to adopt a particular training program. Second, they can be used along with benefit information to calculate a training program's net benefit, benefit-cost ratio (BCR), and ROI. In the next section, we describe how to determine the benefits of training programs.

// The Benefits of Training Programs

The benefits of a training program can be calculated in monetary or non-monetary terms. For example, recall that the benefits of MW Canada's TSCP included an improvement in product quality and a reduction in downtime (non-monetary benefits) and a total annual savings from these improvements of $20,400 (monetary benefit). When the benefit is calculated in monetary terms, it is referred to as a cost-effectiveness evaluation. Acost-effectiveness evaluationinvolves comparing the monetary cost of training to the benefit of training in monetary terms.

Sometimes, however, it is not possible to determine the monetary value of training benefits or to express them in financial terms. Further, in some cases there might be important benefits of a training program that are not monetary benefits. This kind of evaluation is called a cost-benefit evaluation.

Acost-benefit evaluationcompares the cost of training in monetary terms to the benefits of training in non-monetary terms. Non-monetary benefits are similar to what was described as results or Level 4 evaluation criteria in Chapter 10. These benefits involve organization outcomes such as the rate of turnover, absenteeism, customer satisfaction, and so on. It is worth noting that such benefits might have a financial effect on the performance of an organization even though they might not be described in monetary terms.

For example, if a training program is expected to reduce the amount of scrap in the production of a product, then a cost-benefit evaluation would indicate how much the training program cost and the amount or percentage reduction in scrap. On the other hand, a cost-effectiveness evaluation would calculate the monetary value of the reduction in scrap.

Once we know the cost of a training program as well as the benefit in monetary terms, it is possible to determine the net benefit of the program. Thenet benefitof a training program refers to the benefit minus the cost of the training program. Thus, to conduct a net benefit analysis one simply subtracts the cost of a training program from its financial benefit. The net benefit of MW's TSCP is therefore $2,900 ($20,400 $17,500). A related calculation is thebenefit-cost ratio (BCR), which is derived by dividing the benefit by the cost. Thus, the BCR for MW's TSCP is 1.17 ($20,400/$17,500).

Another way to determine the financial benefit of a training program is the calculation of the ROI, which is described in the next section.

// Return on Investment (ROI)

The most popular approach for determining the financial benefits of a training program is return on investment. As indicated in the chapter-opening vignette, MW Canada realized a return on investment of 0.17 or 17 percent.

Return on investment (ROI)involves comparing the cost of a training program to its benefits by dividing the net benefit by the cost of the training program. For example, Cisco Systems calculates the ROI of e-learning by having employees conduct Web-based survey shortly after they have attended a training program. Employees are asked to select a percentage range that indicates the time savings or quality improvement in their performance since taking the course. The results are used to calculate the ROI of e-learning, which has been found to be 900 percent per course. In other words, every dollar the company spends on training results in a gain of $9 in productivity.12

However, many companies in Canada do not calculate the ROI of their training programs. As indicated earlier, only 20 percent of organizations in Canada conduct a Level 5 evaluation. This is due in large part to the difficulty and complexity involved in the process. To help companies learn how to calculate the ROI of their training programs, a program called Investing in People was launched several years ago to provide Canadian organizations with methods and tools to assess the impact of their training investments. To learn more about this program, see Training Today 11.1, "The Investing in People Project."

The calculation of ROI involves dividing the net benefit (benefits - cost of the program) by the cost of a training program:

\mathrm{Return}\;\mathrm{on}\;\mathrm{Investment}=\frac{\mathrm{Benefits}\;-\;\mathrm{Cost}\;\mathrm{of}\;\mathrm{the}\;\mathrm{program}}{\mathrm{Cost}\;\mathrm{of}\;\mathrm{the}\;\mathrm{program}}

ReturnonInvestment=Costoftheprogram

BenefitsCostoftheprogram

As an example, if a training program costs $100,000 and the financial benefit is $300,000, then the calculation of ROI is$300,000 - 100,000/100,000 = 2. In other words, there is a return of $2 for every $1 spent on training (1:2). The percentage return can also be calculated simply by multiplying the ratio by 100; thus, in this case, the return is 200 percent. This can also be described as a 200 percent ROI (the gain of $200,000 is 200 percent of the $100,000 investment).

Returning to MW's TSCP, the ROI is calculated as follows: $20,400 - $17,500/$17,500 = 0.17, or a return of 17 percent (17 100).

TRAINING TODAY 11.1

The Investing in People Project

A project called Investing in People was launched in 2007 to help Canadian organizations evaluate the return on investment (ROI) of training programs and to provide evidence that training results in positive business outcomes. The main objective of the project was to provide evidence that training results in a positive ROI and is crucial to business success and economic growth. The program also aims to encourage Canadian organizations to invest more in training and to identify best training practices based on the most successful training programs.

Investing in People was a three-year project sponsored by Human Resources and Social Development Canada (HRSDC) as part of the Workplace Skills Initiative (WSI). The $1.3-million project was awarded to the Canadian Society of Training and Development (now known as The Institute for Performance and Learning) and was implemented by Learning Designs Online.

ROI case studies were carried out in 12 organizations where a chosen training program was evaluated. The project involved organizations from all geographic regions and includes a wide variety of training programs across the country.

The aim of the program was to develop practical and innovative methods, tools, and instruments for Canadian organizations to use for assessing the impact of their training investments. The program focused on small and medium-sized companies in the manufacturing, services, and retail sectors. The results of the project along with the best practices and evaluation tools have been made available to other Canadian organizations.

To calculate ROI, Learning Designs uses a methodology called the Learning Value Chain, which evaluates a training program at each of four "links" (capability, transfer, business results, and ROI). At each link, data are gathered to assess the extent to which the training has achieved key outcomes, added value, and enabled the next critical event in the chain to occur.

Business Development Bank of Canada (BDC) is one of the 12 workplaces that participated in the program. The Crown corporation has 1700 employees and invests on average about 4.5 percent of its payroll in training. BDC evaluated a coaching program for banking branch managers that achieved an ROI of 74 percent. A training program at WestJet on running efficient meetings achieved an ROI of 558 percent.

Sources: Dobson, S. (2009, October 19). Rallying for ROI around training.Canadian HR Reporter, p. 15; Dobson, S. (2008, April 21). Project connects dots between T&D, profit.Canadian HR Reporter, p. 3; Bailey, A. (2008, Fall). Meeting Canada's productivity challenge.The Canadian Learning Journal, p. 25; Gillis, L., & Bailey, A. (2010).Meta study of evaluation findings. Toronto: Canadian Society for Training and Development.

Let's now return to the wood panel plant. Recall that a supervisor training program that cost $32,564 was designed to improve the quality of wood panels by lowering the daily rejection rate, to improve the housekeeping of the production area, and to reduce the number of preventable accidents.13

Table 11.2 shows how the benefits were measured in each of the three areas. The results in each area before and one year after training are shown, as well as the differences. First, before training, the rejection rate of wood panels was 2 percent per day, or 1,440 panels. After training, this was reduced to 1.5 percent or 1,080 panels. The difference of 0.5 percent per day360 wood panelswas calculated to be a saving of $720 per day or $172,800 per year.

Second, housekeeping was measured in terms of a visual inspection using a 20-item checklist. Before the training there was an average of 10 defects per week; after training it was reduced to two defects. Thus, the training program resulted in a reduction of eight defects per week (this could not be calculated in monetary terms).

Third, the number of preventable accidents before training was 24 per year at a cost of $144,000. After training this was reduced to 16 per year or 8 fewer accidents at a cost of $96,000 and a savings of $48,000.

By comparing this information to the cost information in Table 11.1, we can calculate the net benefit, BCR, and ROI of the training program. Recall that the total cost of the training program was $32,564. The net benefit of the training program in monetary terms can be determined by adding the savings from the reduction in rejected wood panels ($172,800) to the savings from the reduction in preventable accidents ($48,000), and then subtracting the cost of the training program ($32,564). Thus, the net benefit of the training program is$220,800 - $32,564 = $188, 236. The BCR is$220,800/$32,564 = 6.78.

To calculate the ROI, we divide the net benefit of the program ($188,236) by the cost of the training program($32,564) : $188,236/$32,564 = 5.78. Therefore, the ROI for one year after training is equal to 5.78 or 578 percent.

It is worth noting that while this analysis is an example of cost-effectiveness evaluation, the results for housekeeping (i.e., a reduction of eight defects per week) is an example of cost-benefit evaluation.

In summary, this example provides a good illustration of how the benefits of training programs can be measured in a way that is consistent with the objectives of a training program (e.g., reduction in preventable accidents), and can then be translated into monetary terms and used to calculate a training program's net benefit, BCR, and ROI.

Table 11.3 provides a summary of the different methods to calculate the financial benefits of training and development programs using the cost and benefit data from MW's TSCP Training Program. To learn more about how to convert benefits into monetary values, see The Trainer's Notebook 11.2, "Converting Benefits to Monetary Values."

THE TRAINER'S NOTEBOOK 11.2

Converting Benefits to Monetary Values

One of the most difficult aspects of calculating the ROI oftraining and development programs is determining the monetary value of the benefits of training. Jack Phillips, one of the leading experts on the calculation of ROI, suggests the following five steps for converting benefits to monetary values.

Step 1:Focus on a single unit. Identify a particular unit of improvement in output (e.g., products, sales),quality (e.g., errors, product defects), time (to respond to a customer order or finish project), or employee behaviour (e.g., one case of employee turnover).

Step 2:Determine a value for each unit. Place a value identified on the single unit identified in step 1. This will be easier for hard measures such as production, quality, and time because most organizations record the value of one unit of production or the cost of a product defect. It will be more difficult to do for softer measures such as the cost of one employee absence.

Step 3:Calculate the change in performance. Determine the change in performance following training after factoring out other potential influences. This change in units of performance should be directly attributable to the training.

Step 4:Obtain an annual amount. The industry standard for an annual performance change is equal to the total change in the performance data during one year.

Step 5:Determine the annual value. The annual value of improvement equals the annual performance change, multiplied by the unit value.

While the results of this comparison might indicate a significant difference between the two groups, they do not indicate the dollar value associated with the change or improvement in learning, behaviour, or job performance. Utility analysis, however, can work this; it is another approach for determining the financial benefits of training and development programs.

Utility analysisis a method used to forecast the financial benefits that result from human resource programs such as training and development. Utility analysis involves procedures in which the effectiveness of a training program is translated into dollars and cents.14

To calculate the utility of a training program, several factors must be considered. One of the most important is the effectiveness of the training program. In other words, what is the difference in job performance between employees who are trained and those who do not receive training? This difference is called theeffect size. The larger the effect size, the more effective a training program is and the greater its utility.

A second factor is thestandard deviation of job performancein dollars of untrained employees. This has to do with how much of a difference there is in the job performance of untrained employees and the monetary value of this difference.

The standard deviation of job performance in dollar terms is an important factor because in jobs in which the performance of individual employees is widely different, an effective training program will improve the performance of a greater number of employees and will, therefore, result in larger dollar gains. When the job performance of employees is relatively similar, an effective training program is less likely to result in large dollar gains. Therefore, it is necessary to estimate the standard deviation of job performance of untrained employees to make estimates of utility.

There are several approaches for doing this, such as asking supervisors to provide an estimate of the dollar value of performance. The larger the standard deviation of job performance of the untrained group the greater the utility of a training program.

A third factor is thenumber of employees trained. The more employees trained, the greater the utility. A fourth factor is the expected length oftime that the training benefits will last. The longer the effects of training will last, the higher the utility of a training program.

Utility is equal to the multiplication of all of these factors minus thecost of the training program(cost per employee number of employees trained). The following formula is used to estimate the utility of a training program15:

\triangle\mathrm U=\left(\mathrm T ight)\left(\mathrm N ight)\left({\mathrm d}_{\mathrm t} ight)\left(\mathrm{SDy} ight)-\left(\mathrm N ight)\left(\mathrm C ight)

U=(T)(N)(dt

)(SDy)(N)(C)

where:

\mathrm{U}

U= utility, or the dollar value of the program

\mathrm T

T= the number of years the training has a continued effect on performance

\mathrm N

N= the number of people trained

{\mathrm d}_{\mathrm t}

dt

= the true difference in job performance between the average trained and untrained employee in standard deviation units (effect size)

{\mathrm{SD}}_{\mathrm y}

SDy

= the standard deviation of job performance in dollars of the untrained group

\mathrm C

C= cost of training each employee

Consider the following example. To increase the number of toys produced in a toy factory, a training program is implemented and 50 of the plant employees attend. Compared to a group of workers who do not attend the training program, the performance of the 50 trained employees is found to be twice as high (e.g., they produce 100 toys per day compared to 50 produced by untrained workers). We will assume that this equals an effect size of 2. We also assume that the standard deviation of job performance of the untrained employees is $100. The expected length of time that the training will last is estimated to be five years. The cost of the training program is $300 per employee.

Using the utility equation above, we can calculate the utility of the training program as follows:

\begin{array}{ccc}\triangle\mathrm U&=&\left(50 ight)\left(2 ight)\left(\$100 ight)-50\left(\$300 ight)\\\triangle\mathrm U&=&\$50,000\;\$15,\;000\\\triangle\mathrm U&=&\$35,\;000\end{array}

U

U

U

=

=

=

(50)(2)($100)50($300)

$50,000$15,000

$35,000

Thus, the expected utility of the training program for the 50 employees trained is $35,000. This amount might be even greater if the training program lasts longer than five years or if the untrained employees learn how to improve their performance by working with and observing the trained employees. The BCR can also be calculated by dividing the utility by the total cost of the program($35,000/$15,000 = 2.33)and the ROI can be calculated by dividing the net benefit of the program by the total cost of the program($35,000 $15,000/$15,000 = 1.33).

Break-Even Analysis

An extension of the use of the utility formula is to conduct abreak-even analysisor to find the value at which benefits equal costs and utility is equal to zero.16This can be calculated for any of the terms in the utility equation. However, it is most meaningful to conduct a break-even analysis for the effect size or the standard deviation.

For example, what is the break-even effect size for the scenario presented above? This can be calculated by dividing the cost of the training program ($15,000) by the multiplicative function of the other factors:(N)(T)(SD_y)

(N)(T)(SDy

)or (50)(5)(100). The calculations are as follows:

{\mathrm d}_{\mathrm t}=15,\!000/25,\!000

dt

=15,000/25,000

{\mathrm d}_{\mathrm t}=0.6

dt

=0.6

Thus, a training program with an effect size of 0.6 will result in a utility of zero, and an effect size greater than 0.6 will result in a utility that is greater than zero. Therefore, a training program that is considerably less effective than the one in the example would still be likely to result in a financial gain as long as the effect size is greater than 0.6.

Break-even analysis can be very useful because it helps reduce the uncertainty associated with the estimates of the various parameters used to calculate utility. For example, to the extent that the break-even effect size is far below the actual effect size used to calculate utility, the greater the confidence one can have in the results.17

// The Credibility of Benefit Estimates

We have been discussing the costs and benefits of training programs and how to calculate ROI and utility. However, it is important to realize that this is not an exact science. Assumptions and judgments are made when estimating the monetary benefits of a training program. As a result, the process works only if managers and clients accept the assumptions. The estimation of benefits is an inexact procedure and trainers should be concerned about professional credibility.

Credibility is a major issue in cost-effectiveness evaluation, and the data must be accurate and the process believable.18Consider the example of a large bank that was experiencing a high rate of turnover.

A training program was designed to counter the turnover problem. The cost of employee turnover needed to be estimated to calculate the ROI. However, the actual cost calculation was difficult because of the many interacting variablesadministrative costs, interviewing, testing, relocation, orientation, increase in supervisory time, initial less-than-optimal performance, on-the-job trainingthat make up the cost of replacing one person. As the bank did not want to devote the considerable resources necessary to develop a precise calculation, turnover was classified as a soft cost and a combination of approaches was used to derive an acceptable figure.

Initially, a literature search was used to determine that another institution in the same industry had calculated a cost of $25,000 per turnover. This figure, derived by an internal-audit unit and verified by a consulting specialist in turnover reduction, was used as a starting point. The application of this statistic to another (even though quite similar) organization, however, was in question. The training staff then met with senior executives "to agree on a turnover cost value to use in gauging the success of the program. Management agreed on an estimate that was half the amount from the study, $12,500. This was considered very conservative because other turnover studies typically yield statistics of greater value. Management felt comfortable with the estimate, however, and it was used on the benefits side of program evaluation. Although not precise, this exercise yielded a figure that was never challenged."19

The term "never challenged" is significant. Trainers must perform cost-effectiveness evaluations from a position of strength. In this example, senior managers were brought on-side when they were used as experts. It mattered little that the turnover cost was set at $12,500 rather than $25,000, because the benefit estimation produced from these data was credible and accepted by those with the power to make investment decisions.

Thus, despite the appearance of quantitative rigour, virtually all but the simplest cost-effectiveness evaluations are dependent to a greater or lesser extent on some assumptions and expert opinion.20

Trainers must ensure that their clients and management agree on the cost factors and the measurement and estimation of benefits. Management and clients must perceive benefit estimates as credible, believable, and acceptable. It is therefore critical that trainers find out what management deems to be most important in terms of the benefits and expected results and, whenever possible, obtain cost estimates (e.g., the cost of turnover) from management.

It also helps to use internal and external experts to assist in making benefit estimates. Because they are experts who are familiar with the situation, they are likely to be seen as credible by management. For example, if one wanted to estimate the cost of employee grievances, a good expert would be a manager of labour relations. Estimates might also be obtained from other sources that are close to the situation, such as trainees and their supervisors.21

1. If the company wants to conduct a utility analysis, what additional information does it require? How can it obtain this information? In other words, what would DATAIN or the consultant and vendor have to do to obtain the necessary information?
2. What additional information does the company require to conduct a utility analysis? How can it obtain this information? In other words, what would DATAIN or the consultant and vendor have to do to obtain the necessary information?

Explain how to calculate each of the following: net benefit, benefit-cost ratio (BCR), return on investment (ROI), and utility. What are the differences among these calculations?