MIDTERM, ISyE 6416, Spring 2021

Locally weighted linear regression and bias-variance tradeoff.(20 points) Consider a dataset withndata points (xi, yi),xiRp, following the following linear model

yi=Txi+i, i=1,...,n,

wherei N(0,i2) are independent (but not identically distributed) Gaussian noise with zero mean and variancei2.

1

(a) (5 points) Show that the ridge regression which introduces a squaredl2norm penalty on the parameter in the maximum likelihood estimate ofcan be written as follows

() = arg min(Xy)TW(Xy) +2

for property defined diagonal matrixW, matrixXand vectory.

1. (b)(5 points) Find the close-form solution for() and its distribution conditioning on{xi}.
2. (c)(2 points) Derive the bias as a function ofand some fixed test pointx.
3. (d)(3 points) Derive the variance term as a function of.
4. (e)(5 points) Now assuming the data are one-dimensional, the training dataset consists of two samplesx1= 1.5 andx2= 1, and the test samplex= 0.5. The true parameter0= 1,1= 0.5, the noise variance is given by12= 1,2= 3. Plot the MSE (Bias square plus variance) as a function of the regularization parameter.