Structured interviews are some of the most popular measurement devices used for hiring purposes because they tap into managers’ affinity to employment interviews but they also do so in a way that provides structured data that has higher reliability than typical employment interviews. Suppose you are hired as a consultant to Alfaxto Incorporated, a start-up organization that wants to hire a large sales force to push their hot new products. The Vice President for Human Resources (VPHR) who hired you wants you to develop the best possible structured interview system. He says, “I want this to be the scientific state of the art, and so I don’t want you to skip anything in demonstrating its reliability and validity.” The VP-HR is giving you free rein to develop your own structured interview as well as free rein for collecting data to support its use. 

Based on job analyses, you have identified four dimensions that you want to assess in the interview: enthusiasm, verbal ability, problem-solving capability, and goal-driven. You have written a series of ten items per dimension. The VP-HR prides himself on knowing a lot about measurement given that he took a psychological testing course back in college. He wants to know exactly how you are going to evaluate your new structured interview. You try to explain the basic concept of generalizability theory and how you will conduct a GT study along with a D study to figure out the best scenario on how to use this test. He tells you, “Remember, I give you carte blanche to design the best study. On the other hand, I don’t want you wasting my applicants’ time unnecessarily either. Make it just right!” He also says, “I want you to get it right this time around… don’t come back six months later and tell me that we need to include additional factors that weren’t considered this time!” 

Questions

1. What are the factors that you will analyze in your GT study? For each factor, decide whether it is fixed or random. Are any of the factors nested within each other? 

2. What type of D study, absolute or relative, will you conduct to satisfy VP-HR? 

3. From a pragmatic perspective, which factors will be necessary to focus on in the D-study? 

4. VP-HR remembers something about inter-rater reliability from his single testing class. He asks, “Why do we need to do this fancy design? Can’t we just correlate how my ratings compare to yours and be done with it?” How would you answer him? 

5. Do you think the additional effort needed to conduct the GT study is worth it, compared to alternatives?