Question 4

4. Locally weighted linear regression and bias-variance tradeoff. (25 points) Consider a dataset with n data points (mam), 5.3%. 6 RP, following the following linear model M =*T$+Ei' i: 1PM,\"j where 6,; ~ N(O, a?) are independent (but not identically distributed) Gaussian noise with zero mean and variance 0,52. (a) (5 points) Show that the ridge regression which introduces a squared 2 norm penalty on the parameter in the maximum likelihood estimate of [3 can be written as follows 30) = argmgn {(Xb' y)TW(X5 y) + Alllli} for property dened diagonal matrix W, matrix X and vector y. (b (5 points) Find the closeform solution for 30) and its distribution conditioning on {531-}. ) (C) ) ) (5 points) Derive the bias as a function of A and some xed test point 3:. (d (e (5 points) Now assuming the data are onedimensional, the training dataset consists of two samples :51 = 1.5 and 562 = 1, and the test sample a; = 0.5. The true parameter ,6; = 1, {if = 0.5, the noise variance is given by of = 2, 0% = 1. Plot the MSE (Bias square plus variance) as a function of the regularization parameter A. (5 points) Derive the variance term as a function of A