Reflecting for a moment on the results of his ambitious undertaking, Lester smiled. His boss at the Testing and Personnel Services Division of this large midwestern city had assigned him the task of validating the selection test for the job of Supervisor—Children’s Social Worker only two weeks ago, and he had just completed the task. Lester was quite satisfied with the method he’d used to validate the test, and happier still with the results of the effort.

Upon his first inspection of the newly created test, the 145 items seemed to make sense for assessing the behavioral dimensions identified in the job description—knowledge of federal and state laws related to child welfare, supervisory skills, skill in data analytic techniques; skill in reading comprehension; and so on. Still, Lester knew little about the job of a social work supervisor. However, he did have access to the city’s database that contained names of individuals who actually held this position. Through repeated efforts, Lester was able to persuade seven long-time incumbents in the position of Supervisor—Children’s Social Worker to serve as expert raters for assessing the content validity of the new test.

Lester arranged for each of the seven subject matter experts (SMEs) to attend a one-day session. At the beginning of the day, Lester carefully explained the importance of assessing the content validity of the test, and then explained the process that was to be used to review each item on the test. Each rater was to make several independent judgments about each and every item on the test. Specifically, SMEs were asked to consider several dimensions of item QUALITY, including

(a) the appropriate level of difficulty,

(b) the plausibility of item distracters, and

(c) the veracity of the answer key. 

Each SME was then asked to indicate whether he or she was satisfied with all three of these indicators of quality, or not satisfied, if any one of the three characteristics needed improvement. If the latter choice was indicated, the SME was asked to provide additional comments for how the item might be improved.

In addition to the QUALITY rating for an item, each SME provided a second rating based on RELEVANCE. For the relevance ratings, each SME was asked to rate the extent to which the knowledge area or ability assessed by the item was essential to correctly performing the critical functions of the job of Supervisor—Children’s Social Worker. SMEs rated each item as 2 (Essential), 1 (Useful, but not essential), or 0 (Not useful).

Lester then used both the QUALITY and RELEVANCE ratings to determine which items might be kept in the test and which might be deleted or revised. Specifically, Lester decided that items would be retained if the QUALITY rating was “satisfied” by at least 57% of the raters (i.e., four of the seven SMEs) and if the mean RELEVANCE rating for that particular item was at least 1.5. Using these criteria, Lester eliminated 30 items from the original test. The resulting content-validated test retained 115 items.

As a final step, Lester computed the content validity ratio (CVR) for each of the retained 115 items based on the SMEs’ RELEVANCE ratings. Once the CVR was computed for each item, he then computed the mean CVR across all 115 of the retained items. It was this result that pleased Lester the most—the mean CVR score was .78. Reflecting on his work, Lester began to wonder whether now was a good time to ask his boss for a raise.

Questions 

1. Would seven SMEs serve as a sufficient number of expert raters to provide adequate evidence of content validity for this employment selection test? Why or why not?

2. Do the criteria Lester used for inclusion of an item seem appropriate? Defend your response.

3. Why would Lester be happy with a mean CVR rating of .78?

4. What other validation strategies might Lester have employed? What additional information would be needed to adopt a different validation strategy?