4. Locally weighted linear regression and bias-variance tradeoff.

Consider a dataset with n data points (xi , yi), xi R p , following the following linear model yi = T xi + i , i = 1, . . . , n, where i N (0, 2 i ) are independent (but not identically distributed) Gaussian noise with zero mean and variance 2 i .

Show that the ridge regression which introduces a squared l 2 norm penalty on the parameter in the maximum likelihood estimate of can be written as follows () = arg min { (X y) T W(X y) + |||| 2 2

for property defined diagonal matrix W, matrix X and vector y