Variance Estimates Component Estimate 272 060 195 023 FacetS Person (P) Rater (R) Items (I) Var(R) Var(l) Var(P* R) Var(P * I) Var(R*I) Var(Error) 007 443 1. Determine the following values based on the above table (work may be sent to me by e- mail in an Excel sheet) a. What was the mean square error for the 3-way interaction? b. Indicate which facet provides the largest opportunity for improvement; explain in 2 to 3 sentences how you arrived at your answer Would rater training be an effective way to improve the consistency of this measure explain in 2 to 3 sentences? c. d. Calculate the relative error variance for 2 raters and 5 items e. Calculate the absolute error variance for 2 raters and 5 items f. If you were only interested in correctly ordering examinees, what would be the 68% CI around an average essay score of 4 (2 raters and 5 items were used) Calculate the generalizability coefficient for a design with 5 raters and 9 items Calculate the index of dependability (O) for 4 raters and 7 items g. i. Calculate the relative error variance if it was decided that there was no interest in generalizing beyond the 2 raters used to conduct the study (5 items will be administered) Variance Estimates Component Estimate 272 060 195 023 FacetS Person (P) Rater (R) Items (I) Var(R) Var(l) Var(P* R) Var(P * I) Var(R*I) Var(Error) 007 443 1. Determine the following values based on the above table (work may be sent to me by e- mail in an Excel sheet) a. What was the mean square error for the 3-way interaction? b. Indicate which facet provides the largest opportunity for improvement; explain in 2 to 3 sentences how you arrived at your answer Would rater training be an effective way to improve the consistency of this measure explain in 2 to 3 sentences? c. d. Calculate the relative error variance for 2 raters and 5 items e. Calculate the absolute error variance for 2 raters and 5 items f. If you were only interested in correctly ordering examinees, what would be the 68% CI around an average essay score of 4 (2 raters and 5 items were used) Calculate the generalizability coefficient for a design with 5 raters and 9 items Calculate the index of dependability (O) for 4 raters and 7 items g. i. Calculate the relative error variance if it was decided that there was no interest in generalizing beyond the 2 raters used to conduct the study (5 items will be administered)
