In a study examining the effects of individualized care of youths with severe emotional problems, Burchard and Schaefer (1990, personal communication) proposed to have caregivers rate the presence or absence of specific behaviors for each of 40 adolescents on a given day.

To check for rater reliability, they asked two raters to rate each adolescent. The following hypothetical data represent reasonable results for the behavior of “extreme verbal abuse.”

Rater A Rater B Presence Absence Presence 12 2 14 Absence 1 25 26 13 27 40

a. What is the percentage of agreement for these raters?

b. What is Cohen’s kappa?

c. Why is kappa noticeably less than the percentage of agreement?

d. Modify the raw data, keeping N at 40, so that the two statistics move even farther apart.

How did you do this?