Questions and Answers of Measurement Theory In Action

Jaime was rightly proud of the midterm exam he created for the course in which he served as teaching assistant, PSY 451: Introduction to Forensic Psychology. The instructor, Dr. Dan Kellemen, had
“It’ll be easy money.” So said the director of faculty development, when trying to convince Dr. Patricia Lonergan to present a brief overview on test development to a group of interested
Students sometimes complain that items on a test are vague, exceedingly difficult, or even unrelated to the topics presented in the course. Here’s your opportunity to see what it’s like to create
Andrew, a third-year graduate student, was enrolled in a PhD program in quantitative psychology. He had recently obtained a highly competitive summer internship with a Fortune 500 company in its
Linda, a second-year master’s student, had agreed to help out the department of psychology with its outcomes assessment process. In exchange for her work on the project, the department chair agreed
Selected items (13 to be exact) from a 50-item multiple-choice test given to an undergraduate organizational behavior class are presented in Table 13.4. Look through the test to get a sense of the
Alexius, a fifth-year doctoral student, had agreed to sit on the committee that was restructuring the doctoral comprehensive exams for his department. It seemed every year students complained about
Lui-Ping (most people just called her Jasmin), a recent master’s graduate, had decided to return home to Malaysia after graduation. After a short job search, she obtained a job with the ministry of
The Minnesota Multiphasic Personality Inventory (MMPI) is one of the earliest and best-known empirically derived tests. Graham (1977, 1999) presented a detailed description of the development of the
“I knew this topic wasn’t any good, but none of you listened to me, did you?” asked Doug, half in jest. He and four other students in his graduate test construction seminar had recently begun
Use the data file “Joinership rational ratings.sav” to do the following: 1. Compute the mean and standard deviation of each item. 2. Determine a single cut score that you believe reflects
1. Using only those items retained following Exercises 15.2 and 15.3, conduct a reliability analysis of each dimension of the scale (as represented by the factors that emerged in Exercise 15.3).
Ryan, a first-year PhD student, was teaching his own discussion sections of introductory psychology for the first time. At Ryan’s university, the introductory psychology class consists of a
Dora was very excited about her committee’s recent approval of her proposal for her master’s thesis project. After several arduous sets of revisions, she was finally ready to collect her data.
Reflecting concerns that standardized testing was unfair, the psychology department at South East State University (SESU) decided six years ago to eliminate the requirement that prospective students
Although he had worked in human resources (HR) for a little more than a year, Connor Maxfield had been unexpectedly promoted to oversee the selection of new employees into MiniCorp after his boss
Perform a multiple regression analysis using pescore as the dependent variable and each of the six predictors identified in Exercise 8.2 as the independent variables. Choose “enter” as the
Because volunteers are unpaid, the work motivation of these individuals can be complex. It is possible that individuals’ work motivation depends on their perceptions of their own ability, along
Use the data set discussed in Exercise 17.2 to answer the following items. 1. Conduct a multiple regression analysis using all nine predictors. Choose method equals “enter.” Write out the
Andra was on a roll, or at least she had been until she looked at the results of her exploratory factor analysis (EFA). At the request of the chair of the psychology department, she had been working
“Will she ever be happy?” grumbled Chin, mostly to herself. Chin was, of course, referring to her thesis advisor. Everything, it seemed, revolved around her thesis right now. She and her fiancé
In an article published in the Journal of Personality, Marsh (1990) provided evidence that confirmatory factor analysis (CFA) can be very useful in analyzing a multitrait-multimethod (MTMM)
Module 11 discussed the idea of measurement equivalence in the context of cross-cultural research. CFA work has been very important in determining the measurement equivalence of instruments across
Elena, a first-year educational measurement graduate student, vaguely remembered a discussion of item response theory (IRT) in her undergraduate tests and measurements class, but never thought that
Dr. Agars was recently hired by the state of California as a senior research analyst in the Department of Corrections (DoC). His degree was in industrial and organizational (I/O) psychology, with a
1. The Web site http://www.sscicentral.com provides information on several IRT programs, including BILOG-MG, PARSCALE, and IRTPro. The Web site has a student version of IRTPro available for
Scott, a second-year graduate student in educational measurement, had just gotten off the phone with his sister Gail, who had recently finished her nursing degree. Gail and her friend, Tammy, had
The Educational Testing Service identified differential item functioning (DIF) on an analogy question from the SAT exam. The question was an analogy that asked: “Strawberry: Red as (a) peach:
Your advisor wants to develop a new measure of negative affectivity to complement existing measures that are traditional response formats. Your advisor believes that existing measures are limited
Structured interviews are some of the most popular measurement devices used for hiring purposes because they tap into managers’ affinity to employment interviews but they also do so in a way that
Benjamin, a college senior who had a dual major in psychology and marketing, decided he wanted to complete his undergraduate honors thesis in the area of consumer psychology. Specifically, he was
Amanda, a graduate student who had just completed her first year in an industrial and organizational psychology PhD program, was excited because she had just gotten her first consulting job. She was
Outline a scaling study similar to the example that is provided at the end of the overview section of the module. Select a construct (other than school-related social anxiety) and answer the
Using the data from the “Bus Driver.sav” data set, scale the 10 task items on the three dimensions of “Frequency,” “Relative Time Spent,”and “Importance.” Use Table 3.1 to fill in the
Some measures of a particular construct are better than others. Discuss what you believe to be the three most important issues that should be considered prior to item writing that may affect the
As new interns in the human resource department at SAVECO, Juan and Barbara were excited to receive their first major assignment. The president of the company had just asked them to assess the job
The time had come, at last. After countless years as a student, Janie had been teaching her first undergraduate college course—an introductory statistics course—for nearly five weeks. Now the
1. Is the current test related to any other demographic information such as age, education level, or work experience? How about the proposed test?2. The complainant (with ID #450) is suggesting that
1. What descriptive information can we provide to the EEOC regarding the current MC test being used? How about the proposed one?2. Create appropriate graphs to describe the current and proposed MC
Megan, a second-year clinical psychology graduate student, had just gotten her first assignment in her graduate internship placement in the Community Counselling Center (CCC). Dr. Chavez had given
Professor Ullman had just given the first examination in her introductory psychological statistics class. She passed the results on to Rudy, one of her graduate teaching assistants, so that he could
Most tests are administered to identify some form of individual differences. These can include inter-individual differences, intraindividual differences, or both. Again, looking at the Armed Services
As noted in the overview, there are typically numerous constituents in any given testing process. These constituents may include the test takers, test developers, test users, and, more broadly,
The Armed Services Vocational Aptitude Battery (ASVAB) was discussed in the module overview. Nearly one million people take this test each year, many of them high school students. The test consists
A local volunteer referral agency was interested in using “tests” to place volunteer applicants in the volunteer organizations it served. In order to do so, however, the agency needed to assess
Professor Gilbert, an educational testing professor at a local state university, was contacted by a small school district that had decided to implement a Talented and Gifted (TAG) program for
The professor or some other “moderator” will summarize the initial set of ratings for the class. You will then be provided with the results and be allowed to make changes to your initial ratings
“Existential Crisis” of the television show, The Good Place, the characters Tahani and Jason discuss the way Jason rated individuals who auditioned for his dance team. Ratings were conducted on
The “Geoscience attitudes.sav” data set asks students about their attitudes toward the geosciences (archaeology, geography, and geology). The data set has 13 attitude items and 137 respondents.
Extracted communalities and eigenvalues can be computed from an unrotated factor matrix. An extracted eigenvalue is the sum of the squared loadings of the items on a factor. An extracted communality
Imagine you have been asked to develop a personality test that measures antisocial behaviors. The measure will be used for three different purposes. The first purpose will be to identify high school
Find two articles that have used exploratory factor analysis in the development of a psychological scale. Study these two articles, focusing on the documentation of the procedures that were used in
Just as in any other statistical technique, there is a wide variety in terms of the practice and reporting of CFA. Find two studies that use CFA techniques to analyze the structure of a psychological
Either individually or in small groups, perform a literature search to find a recent empirical article that provides an example of the application of IRT to an applied testing situation. IRT
In the module overview, we discussed the key elements of IRT and how it can be applied to computer adaptive testing and differential item functioning. We mentioned that many large-scale tests are now
The Web page https://scholarworks.umass.edu/pare/vol4/iss1/6/ by Ronald Hambleton and H. Jane Rogers is titled “Item Bias Review.” The page provides a number of questions test creators can ask
Either individually or in small groups, perform a literature search to find a recent empirical article that provides an example of the application of IRT to an applied testing situation that
Find two empirical studies that use generalizability theory to evaluate a test or measure (make sure to choose different sets of authors). Study those two articles, paying particular focus on the
One of the leading centers of research on GT has been Center for Advanced Studies in Measurement and Assessment at the University of Iowa, where Dr. Robert Brennan has worked for many years. Check
Assume you wanted to determine the degree of inter-rater agreement between two forensic psychologists who were each rating 100 potential parolees in terms of their potential for committing additional
What do you expect to learn as a student in a psychological testing course?
What will likely be your major stake in the testing process once you finish your measurement course?
What alternative “test” to the APGAR could an obstetrics nurse or doctor use to assess newborn functioning? What would be the advantages and disadvantages compared to the APGAR test shown in
Who are the major constituents or stakeholders in the psychological testing process?
What is the major purpose of examining inter-individual differences via test scores?
What are the different types of intra-individual differences?
What are the major purposes of the different forms of intra-individual differences in interpreting test data?
Can you provide examples of the uses of both inter-individual differences and intra-individual differences?
Are descriptive statistics or inferential statistics used more in applied psychological measurement?
Why are we more likely to use estimation rather than statistical significance testing in applied psychological measurement?
How do descriptive statistics and standardized scores allow us to interpret a set of test scores? Why?
What are the advantages of using a scatterplot in addition to the Pearson product moment correlation?
What does a 95% confidence interval of the mean tell us? How about a 99% confidence interval for an individual score?
What is the difference between scaling and classification?
What is the difference between psychometrics and psychological scaling?
Why do you think it is so difficult to scale more than one dimension (i.e., people, stimuli, and responses) at once?
Why is it important to know the level of measurement of our data before we begin the scaling process?
How would we scale multiple dimensions at one time?
Why do measures that claim to assess the same construct sometimes appear so vastly different from one another?
To what extent is it likely that two different measures of the same construct that employ the use of distinct item formats will provide similar results?
What practical constraints often play a large role in the determination of test specifications?
If you were assigned to develop a new measure for a personality construct, what sources might you seek to better inform yourself about the construct prior to defining the construct?
When developing a measure intended to assess a facet of personality, when would you develop items that use frame-of-reference tags? When would you develop items that assess behavior across contexts?
Why is reliability important in psychological and educational testing?
In your own words, explain the concept of a true score.
What are some of the major assumptions of classical test theory?
What are some of the limitations of classical test theory?
How is reliability defined in terms of classical test theory?
Under what conditions might we want a very high reliability coefficient?
Under what conditions might we accept a low reliability coefficient for a psychological measure?
Sheila was frustrated. Although she was happy with both the topic and the constructs she had chosen to examine in her senior honors thesis, she had hit several roadblocks in determining what measures
Why are longer tests generally more reliable than shorter tests? What conditions must be met for this to be true?
Validity is a unified construct. In what ways do the various approaches to examining content validity provide validity evidence?
Would it be more appropriate to adopt a content validation approach to examine a final exam in a personality psychology course or to examine a measure of conscientiousness? Explain.
The content approach to test validation has historically relied heavily on expert judgment, though newer approaches argue that any competent individual might be capable of providing validity evidence
Would content validity alone provide sufficient evidence for validity for(a) An employment exam?(b) An extraversion inventory?(c) A test to determine the need for major surgery? In each case,
Does face validity establish content validity? Explain your answer.
What could a student do if he or she thought a classroom exam was not content valid?
What could an instructor do if a student asserted that a classroom exam was not content valid?
Consider a test or inventory of your choosing. If you wanted to examine the content validity of this measure, how would you go about choosing raters to provide judgments regarding the content

Showing 1000 - 1100 of 1226