Class Exercise on Test Validity

Instructions: For this activity you will be reviewing the results of a new personality test and a new interview process that have been developed as part of the selection process for a sales position at ABC Sales. The correlation matrix (on page 2) shows the correlations between each part of the application process and job performance. For reference, the bottom row shows the correlation between each piece of information the company has about candidates and job performance (as rated by supervisors). 1-6 are pieces of information obtained about the candidate during the job application process. 7 is supervisor ratings of job performance 4 months post-hire.

How to read the correlation matrix: As an example for reading the correlation matrix, the correlation between the new extraversion assessment (#3) and job performance (#7) is .198. The correlation between the original extraversion assessment (#2) and the new extraversion assessment (#3) is .650.

Information from the staffing process:

Years of experience in sales: This could be considered a biodata item which you will read about in chapter 9, it is also something that is included in almost all job application processes.

Personality: Two personality traits are assessed that could be important for sales Extraversion (e.g., outgoing, sociable) and Conscientiousness (e.g., achievement striving and detail oriented). To shorten the application process, a new extraversion assessment was included that only uses 4 questions. The original extraversion assessment used 10 questions. The new questions are also framed as being at work.

Original Interview: The original interview process included a somewhat informal panel interview. The managers were not asked to score individual questions, but each panel member scored the overall interview, and each manager could ask any questions they wanted to.

Structured Interview: To improve their hiring process, ABC sales developed a new interview process. The new questions are more structured meaning everyone gets the same set of questions. Each panel member then scores each question and is being provided with a rubric/guideline to score each response.

Job Performance: This indicates supervisor ratings of job performance on a 1-10 scale.

Reliability of each piece of information: The bold numbers on the diagonal line are the reliability of each type of assessment/piece of information. For the years of experience the correlation is 1.0 because someones experience is just a single number therefore it correlates perfectly with itself. For assessments #2-4 and job performance (#7) the reliability refers to inter-item reliability. For the interviews (#5-6) reliability refers to inter-rater reliability from panel interviews. Inter-rater reliability is often somewhat lower than other types of reliability, so the lower reliability numbers from the interviews are not entirely surprising.

Correlation Matrix

	1	2	3	4	5	6	7
1. Years of experience in sales	(1.0)[GB1]
2. Original Extraversion assessment	.059	(.65)
3. New Extraversion assessment	.120	.650	(.81)
4. Conscientiousness assessment	.150	-.025	-.056	(.82)
5. Original Interview	.325	.275	.215	.150	(.60)
6. New Structured Interview	.257	.125	.195	.185	.620	(.68)
7. Job Performance	.145	.109	.175	.245	.275	.395	(.72)

After looking over the correlation matrix, answer the following questions.

1. Based on the available information, which pieces of information/assessments would you recommend using for the selection process? Explain why or why not.

2. What additional information would you want to know to help you make a recommendation, and how would that information influence your recommendations?