PSYC 421 ITEM DEVELOPMENT AND Student Name: ANALYSIS WORKSHEET Section: PSYC421- LaTarsha Jones D02 PART 1: Writing Multiple Choice Test Items (Cohen et al., 2013, pg. 252) Develop one multiple choice question that covers content from each of the four chapters listed below. When writing your sample questions, please keep in mind the specifications regarding item construction discussed in the textbook. Also, remember the importance of carefully crafted distractor options. Finally, please limit the number of response options to 4 (1 correct response and 3 distractors), and avoid the options of \"all of the above,\" none of the above,\" or the like. Be sure to indicate which of the response options is the correct one. Chapter 3 Multiple Choice Question (2.5 points) Chapter 4 Multiple Choice Question (2.5 points) Chapter 5 Multiple Choice Question (2.5 points) Chapter 6 Multiple Choice Question (2.5 points) Page 1 of 7 PSYC 421 PART 2: Item Analysis: Item Difficulty Index (Cohen et al., 2013, pg. 263) A test is only as good as its questions! When researchers, test constructors, and educators create items for ability or achievement tests, we have a responsibility to evaluate the items and make sure that they are useful and high-quality. The process that we use to evaluate test items is known as Item Analysis. When bad items are identified and eliminated from a test, that increases the efficiency, reliability and validity of the entire test! One way that we can distinguish among good and bad items is with the Item Difficulty Index. Part 2A: Calculating Item Difficulty Using the data below, calculate the Item Difficulty Index for the first 6 items on Quiz 1 from a recent section of PSYC101. For each item, \"1\" means the item was answered correctly and \"0\" means it was answered incorrectly. Type your answers in the spaces provided at the bottom of the table. (1 pt. each) PSYC101 Quiz 1 Item Distribution and Total Scores Examinee Item 1 Item 2 Item 3 Item 4 Item 5 Item 6 Total Score Andre 1 1 1 1 1 1 16 Allison 1 1 1 1 0 0 7 Heather 1 1 1 1 0 0 10 Corey 1 1 0 1 1 1 17 Christina 0 0 1 1 0 1 3 Jeffrey 0 1 1 1 0 0 11 Shawn 1 1 1 1 0 1 14 Dana 0 0 1 1 0 1 10 Megan 1 1 1 1 0 1 13 David 0 1 1 1 0 1 12 Isabel 0 0 0 1 0 0 4 Lance 1 1 1 1 0 0 9 Aliyah 1 1 1 1 0 1 15 Blaire 0 1 1 1 0 1 12 Gabriel 0 0 1 1 0 0 6 Page 2 of 7 PSYC 421 Item Difficulty Part 2B: Calculating Optimal Item Difficulty (.5 pt. each) 1. For a test item with two response options (e.g., true/false), what is the probability of selecting the correct answer by chance? % 2. Calculate the optimum level of difficulty for a test questions with two response options. 3. For a test item with three response options, what is the probability of selecting the correct answer by chance? % 4. Calculate the optimum level of difficulty for a test questions with three response options. 5. For a test item with four response options, what is the probability of selecting the correct answer by chance? % 6. Calculate the optimum level of difficulty for a test questions with four response options. 7. For a test item with five response options, what is the probability of selecting the correct answer by chance? % 8. Calculate the optimum level of difficulty for a test questions with five response options. PART 3: Item Analysis: Item Discrimination Index (Cohen et al., 2013, pg. 265-266) Another way that test creators can distinguish between good and bad items is with an analysis called the Discrimination Index. The discrimination index measures how well an individual test item distinguishes between high scorers and low scores on the test. An item is considered to be \"good\" if most of the high scorers get it right, and most of the low scorers get it wrong. Interpreting the Discrimination Index (d) Page 3 of 7 PSYC 421 The discrimination index can range from -1.0 to 1.0. The closer d is to 1.0, the better the item discriminates between high and low scorers The closer d is to 0, the more poorly the item discriminates between high and low scorers. An item with a negative discrimination index is considered a \"negative discriminator\" because more low scorers get the item correct than high scorers. A discrimination index of 1.0 means all the high scorers got the item correct and all of the low scorers got it incorrect. A discrimination index of -1.0 means all of the low scorers got the item correct and all of the high scorers got it incorrect. Items with d's close to 0 or with negative d's ought to be eliminated from the test! Calculating the Item Discrimination Index (d) Calculate the item discrimination index (d) for the 7 hypothetical test items presented below. Type your answers in the spaces provided at the right of the table (1 pt. each). Item # U L n Item 1 21 17 25 Item 2 23 7 25 Item 3 25 0 25 Item 4 3 24 25 Item 5 22 3 25 Item 6 0 25 25 Item 7 19 6 25 d Based on your calculations above, answer the following questions (1 pt. each). 1. Which item discriminates the best? 2. Which item discriminates most poorly? 3. Based on your analysis, identify which two items would you choose to eliminate from this test and explain why you would eliminate each. Page 4 of 7 PSYC 421 Part 4: Item Characteristic Curves (Cohen et al., pg. 268-270) Another method that test creators can use to assess the usefulness of test items is with Item Characteristic Curves. Item characteristic curves provide a graphical depiction of examinees' performance on individual test items. As indicated in the figure below, Total Test Score is plotted on the x-axis of the graph, while proportion of examinees who got the item correct is plotted on the y-axis. Using the figure above, provide a written description of how test items A-D discriminate among examinees at various levels of performance. In your responses, discuss why each item would be considered a \"good\" or a \"bad\" item. EXAMPLE: \"This item discriminates well among high scores, but doesn't discriminate well among low scorers. So this item would be considered a good item because it discriminates at the highest levels of performance.\" (2 pt. each) Item A: Item B: Page 5 of 7 PSYC 421 Item C: Item D: Item E: Part 5: Qualitative Item Analysis (Cohen et al., pg. 272-274) Qualitative item analysis refers to a set of non-statistical procedures used to gather information about the usefulness of test items. These analyses typically involve interviews, panel discussions, questionnaires and other forms of verbal exchange with test-takers to explore how individual test items work. As an online student, you have a very different test-taking experience than residential students. Based on your readings from Chapter 8, identify 4 topics related to online test taking, and create 4 qualitative questions that you could ask online test-takers to gain an understanding of their experiences with testtaking. Also, as students at a Christian institution of higher education, course assignments/assessments are supposed to give students an opportunity to integrate course content with their Christian worldview. Given the topic of faith and learning, create one qualitative question that you could ask test-takers. Topic (1 pt. each) Sample Question for Test-Takers (1 pt. each) Page 6 of 7 PSYC 421 Assignment Scoring (Instructor Use Only) Part 1 Subtotal: /10 Part 2 Subtotal: /10 Part 3 Subtotal: /10 Part 4 Subtotal: /10 Part 5 Subtotal: /10 TOTAL SCORE: /50 Page 7 of 7 Chapter 3 A Statistics Refresher 77 SCALES OF MEASUREMENT 78 Nominal Scales 79 Ordinal Scales 80 Interval Scales 81 Ratio Scales 81 Measurement Scales in Psychology 82 DESCRIBING DATA 83 Frequency Distributions 83 Measures of Central Tendency 89 Measures of Variability 92 Skewness 96 Kurtosis 97 THE NORMAL CURVE 98 The Area Under the Normal Curve 99 STANDARD SCORES 102 z Scores T Scores 103 Other Standard Scores 103 CORRELATION AND INFERENCE 106 The Concept of Correlation 106 The Pearson r 107 The Spearman Rho 110 Graphic Representations of Correlation 111 Meta-Analysis 115 EVERYDAY PSYCHOMETRICS 88 CLOSE-UP Consumer (of Graphed Data), Beware! The Normal Curve and Psychological Tests 100 MEET AN ASSESSMENT PROFESSIONAL Meet Dr. Benoit Verdon 116 SELF-ASSESSMENT Chapter 4 Of Tests and Testing 118 SOME ASSUMPTIONS ABOUT PSYCHOLOGICAL TESTING AND ASSESSMENT Assumption 1: Psychological Traits and States Exist Assumption 2: Psychological Traits and States Can Be Quantified and Measured Assumption Assumption 3: Test-Related Behavior Predicts Non-Test-Related Behavior Assumption 4: Tests and Other Measurement Techniques Have Strengths and Weaknesses Assumption 5: Various Sources of Error Are Part of the Assessment Process Assumption 6: Testing and Assessment Can Be Conducted in a Fair and Unbiased Manner Assumption 7: Testing and Assessment Benefit Society 123 WHAT'S A \"GOOD TEST\"? 124 Reliability 124 Validity 125 Other Considerations 125 NORMS 128 Sampling to Develop Norms 129 Types of Norms 135 Fixed Reference Group Scoring Systems 139 Norm-Referenced Versus Criterion-Referenced Evaluation 139 CULTURE AND INFERENCE CULTURE AND INFERENCE 142 EVERYDAY PSYCHOMETRICS Putting Tests to the Test 126 CLOSE-UP 130 How \"Standard\" Is Standard in Measurement? MEET AN ASSESSMENT PROFESSIONAL Meet Dr. Steve Julius and Dr. Howard W. Atlas SELF-ASSESSMENT 143 Chapter 5 142 5 Reliability 145 THE CONCEPT OF RELIABILITY 145 Sources of Error Variance 147 RELIABILITY ESTIMATES 150 Test-Retest Reliability Estimates 150 Parallel-Forms and Alternate-Forms Reliability Estimates 151 Split-Half Reliability Estimates 152 Other Methods of Estimating Internal Consistency 154 Measures of Inter-Scorer Reliability 159 USING AND INTERPRETING A COEFFICIENT OF RELIABILITY The Purpose of the Reliability Coefficient The Nature of the Test 160 160 161 The True Score Model of Measurement and Alternatives to It RELIABILITY AND INDIVIDUAL SCORES 164 173 The Standard Error of Measurement 175 The Standard Error of the Difference Between Two Scores 178 EVERYDAY PSYCHOMETRICS The Reliability Defense and the Breathalyzer Test CLOSE-UP Item Response Theory (IRT) 170 MEET AN ASSESSMENT PROFESSIONAL SELF-ASSESSMENT 180 Meet Dr. Bryce B. Reeve 174 165 PSYC 421 ITEM DEVELOPMENT AND Student Name: ANALYSIS WORKSHEET Section: PSYC421- LaTarsha Jones D02 PART 1: Writing Multiple Choice Test Items (Cohen et al., 2013, pg. 252) Develop one multiple choice question that covers content from each of the four chapters listed below. When writing your sample questions, please keep in mind the specifications regarding item construction discussed in the textbook. Also, remember the importance of carefully crafted distractor options. Finally, please limit the number of response options to 4 (1 correct response and 3 distractors), and avoid the options of \"all of the above,\" none of the above,\" or the like. Be sure to indicate which of the response options is the correct one. Chapter 3 Multiple Choice Question (2.5 points) Chapter 4 Multiple Choice Question (2.5 points) Chapter 5 Multiple Choice Question (2.5 points) Chapter 6 Multiple Choice Question (2.5 points) Page 1 of 7 PSYC 421 PART 2: Item Analysis: Item Difficulty Index (Cohen et al., 2013, pg. 263) A test is only as good as its questions! When researchers, test constructors, and educators create items for ability or achievement tests, we have a responsibility to evaluate the items and make sure that they are useful and high-quality. The process that we use to evaluate test items is known as Item Analysis. When bad items are identified and eliminated from a test, that increases the efficiency, reliability and validity of the entire test! One way that we can distinguish among good and bad items is with the Item Difficulty Index. Part 2A: Calculating Item Difficulty Using the data below, calculate the Item Difficulty Index for the first 6 items on Quiz 1 from a recent section of PSYC101. For each item, \"1\" means the item was answered correctly and \"0\" means it was answered incorrectly. Type your answers in the spaces provided at the bottom of the table. (1 pt. each) PSYC101 Quiz 1 Item Distribution and Total Scores Examinee Item 1 Item 2 Item 3 Item 4 Item 5 Item 6 Total Score Andre 1 1 1 1 1 1 16 Allison 1 1 1 1 0 0 7 Heather 1 1 1 1 0 0 10 Corey 1 1 0 1 1 1 17 Christina 0 0 1 1 0 1 3 Jeffrey 0 1 1 1 0 0 11 Shawn 1 1 1 1 0 1 14 Dana 0 0 1 1 0 1 10 Megan 1 1 1 1 0 1 13 David 0 1 1 1 0 1 12 Isabel 0 0 0 1 0 0 4 Lance 1 1 1 1 0 0 9 Aliyah 1 1 1 1 0 1 15 Blaire 0 1 1 1 0 1 12 Gabriel 0 0 1 1 0 0 6 Page 2 of 7 PSYC 421 Item Difficulty 0.5333 0.7333 0.8667 1.0000 0.1333 0.6000 Part 2B: Calculating Optimal Item Difficulty (.5 pt. each) 1. For a test item with two response options (e.g., true/false), what is the probability of selecting the correct answer by chance? % 75 2. Calculate the optimum level of difficulty for a test questions with two response options. 0.75 3. For a test item with three response options, what is the probability of selecting the correct answer by chance? % 67 4. Calculate the optimum level of difficulty for a test questions with three response options. 0.67 5. For a test item with four response options, what is the probability of selecting the correct answer by chance? % 63 6. Calculate the optimum level of difficulty for a test questions with four response options. 0.63 7. For a test item with five response options, what is the probability of selecting the correct answer by chance? % 60 8. Calculate the optimum level of difficulty for a test questions with five response options. 0.60 PART 3: Item Analysis: Item Discrimination Index (Cohen et al., 2013, pg. 265-266) Another way that test creators can distinguish between good and bad items is with an analysis called the Discrimination Index. The discrimination index measures how well an individual test item distinguishes between high scorers and low scores on the test. An item is considered to be \"good\" if most of the high scorers get it right, and most of the low scorers get it wrong. Interpreting the Discrimination Index (d) Page 3 of 7 PSYC 421 The discrimination index can range from -1.0 to 1.0. The closer d is to 1.0, the better the item discriminates between high and low scorers The closer d is to 0, the more poorly the item discriminates between high and low scorers. An item with a negative discrimination index is considered a \"negative discriminator\" because more low scorers get the item correct than high scorers. A discrimination index of 1.0 means all the high scorers got the item correct and all of the low scorers got it incorrect. A discrimination index of -1.0 means all of the low scorers got the item correct and all of the high scorers got it incorrect. Items with d's close to 0 or with negative d's ought to be eliminated from the test! Calculating the Item Discrimination Index (d) Calculate the item discrimination index (d) for the 7 hypothetical test items presented below. Type your answers in the spaces provided at the right of the table (1 pt. each). Item # U L n d Item 1 21 17 25 0.16 Item 2 23 7 25 0.64 Item 3 25 0 25 1 Item 4 3 24 25 - 0.84 Item 5 22 3 25 0.76 Item 6 0 25 25 -1 Item 7 19 6 25 0.52 Based on your calculations above, answer the following questions (1 pt. each). 1. Which item discriminates the best? item 5 2. Which item discriminates most poorly? item 6 3. Based on your analysis, identify which two items would you choose to eliminate from this test and explain why you would eliminate each. Page 4 of 7 PSYC 421 Item 4 and item 6 ought to be eliminated from the test since they discriminte negatively. Part 4: Item Characteristic Curves (Cohen et al., pg. 268-270) Another method that test creators can use to assess the usefulness of test items is with Item Characteristic Curves. Item characteristic curves provide a graphical depiction of examinees' performance on individual test items. As indicated in the figure below, Total Test Score is plotted on the x-axis of the graph, while proportion of examinees who got the item correct is plotted on the y-axis. Using the figure above, provide a written description of how test items A-D discriminate among examinees at various levels of performance. In your responses, discuss why each item would be considered a \"good\" or a \"bad\" item. EXAMPLE: \"This item discriminates well among high scores, but doesn't discriminate well among low scorers. So this item would be considered a good item because it discriminates at the highest levels of performance.\" (2 pt. each) Item A: This item discriminates equally among high scores and low scores so this item is considered good Item B: This item discriminates negatively among high scores and low scores. This item therefore is not the Page 5 of 7 PSYC 421 Item C: This item discriminates well among high scores but does not discriminate well among low scores. T Item D: This item discriminates fairly the same on the high scores as it discriminates the low scores. This ite Item E: This item discriminates well among low scores than the high cores. This item is not the best discrim Part 5: Qualitative Item Analysis (Cohen et al., pg. 272-274) Qualitative item analysis refers to a set of non-statistical procedures used to gather information about the usefulness of test items. These analyses typically involve interviews, panel discussions, questionnaires and other forms of verbal exchange with test-takers to explore how individual test items work. As an online student, you have a very different test-taking experience than residential students. Based on your readings from Chapter 8, identify 4 topics related to online test taking, and create 4 qualitative questions that you could ask online test-takers to gain an understanding of their experiences with testtaking. Also, as students at a Christian institution of higher education, course assignments/assessments are supposed to give students an opportunity to integrate course content with their Christian worldview. Given the topic of faith and learning, create one qualitative question that you could ask test-takers. Topic (1 pt. each) Sample Question for Test-Takers (1 pt. each) Page 6 of 7 PSYC 421 Assignment Scoring (Instructor Use Only) Part 1 Subtotal: /10 Part 2 Subtotal: /10 Part 3 Subtotal: /10 Part 4 Subtotal: /10 Part 5 Subtotal: /10 TOTAL SCORE: /50 Page 7 of 7 Part 1: Questions for each chapter Chapter 3 question: Five students scored the following marks in Mathematics test 60 65 70 55 40 82 What is the mean mark score of the class? A. B. C. D. 74.5 74 60 82 (Correct answer is B) Chapter 4 Question The table below shows the number of students in a school and games the play. Game Number of students Football 13 Hockey 16 Table tennis 17 Volley ball 5 Which of the games in the table are you likely to get a chance to play if you join. A. B. C. D. Table tennis: Because many students like the game Football: Because the game is famous and easily played Volleyball: Because less student like the game and there no enough players Hockey: Because the game has a sizeable number of players. (Correct answer is C) Chapter 5 Question A five subject test was administered as a mandatory for admission in some private school. The subjects considered were Mathematics, English, General Science, Social Science and History. After the test the subjects had the following reliability coefficients: Subject Reliability coefficient Mathematics 0.6 English 0.4 General Science 0.8 Social Science 0.7 History 0.9 Which subject was poorly done? A. B. C. D. History English Mathematics General Science (Correct answer is B) Part 5 creating online test questions Online test taking topics: 1. 2. 3. 4. Measures of central tendencies Measures of variability Measures of correlation Measures of item difficulty Online test taking questions (Click the correct answer) 1. The following marks were scored in a class of 15 students. Calculate the range. 25, 26, 15, 10, 47, 36, 12, 10, 50, 21, 16, 30, 42, 15, 51 o 40 o 41 o 12 o 10 2. From the data obtained in question 1, what is the standard deviation? o 14.82 o 15.0 o 27.2 o 41.0 3. A Mathematics teacher wanted to know the correlation in the scores in her subject between girls and boys and decide on how to form remedial groups for the learners. The following scores were obtained from a previous Mathematics test. Scores for boys 12 15 30 55 65 30 75 72 45 36 45 65 62 52 Scores for girls 17 26 51 63 22 16 65 70 43 34 55 61 70 46 What is the coefficient of correlation? o 0.071 o 0.710 o 0.701 o 0.070 4. In an aptitude test, the teacher decided to ask a random question to a class of 25 students. If 13 students scored the wrong. What is the item difficulty index? o 0.52 o 0.48 o 1.0 o 0.61 A question of faith on learning Which among the list below is NOT the correct way of remaining helpful to our friends? o o o o Praying for them when they are in difficulties Doing assignments for them Lending them pens and books in class when they haves lost theirs Inviting them for praise and worship