Olivia and Gavin had been studying for their upcoming psychometrics midterm for over an hour, but the past few minutes had been less than productive. While the two thought they both had a good understanding of the various types of reliability, they just couldn’t agree on when each measure should be used.

“Let’s say we were going to test military officers to determine who might be best fit for a promotion. Which form of reliability would we be most concerned with?” asked Gavin.

Lauren thought for a brief moment before responding, “Assuming there were a lot of officers to test, I’d bet the most serious concern would be with the officers sharing information about the test with others. Since test security would be a big concern, they’d want to develop more than one form of the test, but they’d want to be certain that each form was equivalent. I’d say, then, that parallel forms reliability would be the type of reliability they’d be concerned with.”

“That’s what I was thinking,” said Gavin, “but when is coefficient alpha going to be used?”

Olivia was ready for that, “I think coefficient alpha is likely to be a major consideration when we want to ensure that the items on the test are highly interrelated. That’s a primary concern whenever we have paper-and-pencil–based measures of psychological constructs like intelligence, extraversion, or even job satisfaction.”

“You always know everything,” said Gavin, “That’s why I like studying with you. You are so darned conscientious.”

“I wish I did,” said Olivia, “but I am having a hard time coming up with an example of when to use test-retest reliability.”

“Actually,” said Gavin, “I think you just helped me come up with an example. Your conscientiousness, like any other personality variable, is supposed to be pretty stable, right? So our measures of those variables should be too.”

Olivia beamed, “Right. So if our measure of a personality construct like conscientiousness yields about the same score over repeated administrations, then we’d feel more confident about the measure.”

“Exactly,” stated Gavin. “Ready to move on to the next topic?” Olivia glanced at her watch and frowned. “I wish I could, but I’ve got to go. I promised Dr. Warren I’d help him judge the student research poster competition this afternoon.”

“Alright,” said a clearly disappointed Gavin, “but I sure hope you and Dr. Warren have good inter-rater reliability!”

Questions

1. Why is coefficient alpha such a commonly reported reliability estimate in psychology and education?

2. Provide additional examples of instances in which each type of reliability (test-retest, parallel forms, internal consistency, and inter-rater reliability) might be used.

3. In judging the reliability of judges’ ratings of a student research competition, would we be satisfied with inter-rater reliability as computed by a correlation coefficient, or would computation of kappa be necessary?

4. Are there times when we might be interested in obtaining more than one type of reliability? Explain by providing an example.