“This will be a cinch,” Cecilia had thought when she first received the assignment to conduct a criterion-related validity study. She’d been a human resource (HR) specialist at Joyco for only three weeks, and she relished the thought of tackling her first major project independently. In fact, she had jumped at the opportunity when her boss asked her to conduct a criterion-related validation study to determine whether a newly created test of mechanical ability would be useful in selecting production workers. She thought it would be relatively easy to collect test scores and correlate them with some measure of job performance. Only slowly did she realize how much thought and hard work would actually take place to complete the task properly.

She soon realized the first major issue she’d have to tackle was identifying an appropriate criterion. Clearly, job performance was appropriate, but how should job performance be measured? Supervisors formally appraised each production worker’s performance annually, and the HR office seemed pleased with the quality of the process. Still, Cecilia knew the supervisor ratings were far from perfect assessments of an employee’s job performance.

Cecilia’s thoughts suddenly leaped to another concern—when her boss had asked her to determine the criterion-related validity of the new test, she had provided a two-week deadline. Such a tight deadline clearly precluded use of a predictive design. Unfortunately, the current production workers were likely very different from job applicants. In comparison to applicants, current production workers tended to be older, they were more similar to one another ethnically, and they also had much more job experience. Given the timeline provided by her boss, however, Cecilia thought that it was the current production workers that she’d need to use to validate the proposed selection tests.

Undeterred, Cecilia went ahead with the project. Cecilia administered the new mechanical ability test to a sample of Joyco’s production workers. In an effort to ensure she’d done a complete job, Cecilia also collected information on everything she thought might be relevant. Nonetheless, she knew she had collected an impressive amount of information regarding the new test of mechanical ability and the criterion, including the following:

Sample size for the validation study = N = 178 job incumbents Observed validity = r_xy = .24 

Reliability of the new mechanical ability test = r_xx = .85 

Inter-rater reliability of supervisor ratings of job performance = r_yy = .78 

Standard deviation of mechanical ability tests scores for current employee sample = 9 

Standard deviation of mechanical ability test scores for applicant sample = 14 

Although the completion deadline was quickly approaching, Cecilia still had a ways to go before producing an accurate criterion-related validity estimate for the new mechanical ability test. Cecilia slumped back into her chair. “This is definitely going to take some work,” she thought.

Questions 

1. What research design did Cecilia use to conduct her criterionrelated validation study? What was the major determinant of this decision?

2. Why might Cecilia have preferred another research design for her criterion-related validation study?

3. Would Cecilia have to be concerned with criterion contamination in conducting this validation study? Explain.

4. Identify three alternate criteria Cecilia might have used to assess job performance, rather than supervisor ratings. What concernsdo you have with each possible criterion?

5. Given the sample used to validate the proposed selection tests, which correction formulas would be most important to use?

6. Given the data Cecilia collected, compute each of the following:

a. Correction for attenuation in the criterion

b. Correction for predictor range restriction.

7. Examining the corrected validity coefficient produced, how impressed would you be with this test of mechanical ability for use in selection of new employees? Would you recommend use of the test? Explain.