1. Consider a situation with one response variable Y and one predictor variable X which are related according to the following statistical model Y =1+X+ (1) where e~ N(0,1) is a Gaussian random variable with zero mean and unit variance. a) What is the regression function f(x)? b) A training set is a set of three ordered pairs(x, y) for x = 1,2,3. Create a training set by generating random variable Y according to the above equation for X = 1,2,3. Fit three different models to the training data Mo, M1, M2 given by Mo : Y = Bo (2) M : Y = Bo + BiX (3) M2 : Y = Bo +81X + B2X2 (4) and observe the training error for the three models. What is the training error for model M? Is it consistent with what you expected? c) Now generate 1000 different training sets and let Tri = {(1, yli), (2, yzi), (3, y3i)} denote the ith training set. Now, you have a set of 1000 training sets. For each training set Tri in the set of training sets, fit each of the three models above and record the model parameters ( s), training error and predict the value of Y for X = 5. Let the prediction based on the ith training set for model M; be fuj(5). d) Let Mj := 1000 L; fij. For each j compute the square of the bias given by (f(5) M;)2 where f(5) is the regression function evaluated at X = 5, the variance of f;(5) and the variance of the irreducible error. Plot the squared bias, variance of f;(5) and the variance of the irreducible error as a function of model complexity j. 1 e) Compute the expected value of the testing error for each training set Vij = E[Y(5) - fij (5)]2 and average it over i to compute the expected value of testing error for each model Mo, M1 and M2. This gives you the expected value of the testing error if the training set and the value for Y and X = 5 are drawn from the distribution in (1). f) Verify equation (2.7) based on these calculations. 1. Consider a situation with one response variable Y and one predictor variable X which are related according to the following statistical model Y =1+X+ (1) where e~ N(0,1) is a Gaussian random variable with zero mean and unit variance. a) What is the regression function f(x)? b) A training set is a set of three ordered pairs(x, y) for x = 1,2,3. Create a training set by generating random variable Y according to the above equation for X = 1,2,3. Fit three different models to the training data Mo, M1, M2 given by Mo : Y = Bo (2) M : Y = Bo + BiX (3) M2 : Y = Bo +81X + B2X2 (4) and observe the training error for the three models. What is the training error for model M? Is it consistent with what you expected? c) Now generate 1000 different training sets and let Tri = {(1, yli), (2, yzi), (3, y3i)} denote the ith training set. Now, you have a set of 1000 training sets. For each training set Tri in the set of training sets, fit each of the three models above and record the model parameters ( s), training error and predict the value of Y for X = 5. Let the prediction based on the ith training set for model M; be fuj(5). d) Let Mj := 1000 L; fij. For each j compute the square of the bias given by (f(5) M;)2 where f(5) is the regression function evaluated at X = 5, the variance of f;(5) and the variance of the irreducible error. Plot the squared bias, variance of f;(5) and the variance of the irreducible error as a function of model complexity j. 1 e) Compute the expected value of the testing error for each training set Vij = E[Y(5) - fij (5)]2 and average it over i to compute the expected value of testing error for each model Mo, M1 and M2. This gives you the expected value of the testing error if the training set and the value for Y and X = 5 are drawn from the distribution in (1). f) Verify equation (2.7) based on these calculations