(a) If n > p, the estimated parameter of the regression model y = x + is = (XX)1Xy, where X Rnp and y Rn are the data, and is a zero mean random variable with var() = 2. Note that Rp is a random vector as it is function of the data. Show that var( ) = 2(XX)1.

(b) When n p, we need to use an optimization algorithm, e.g., Gradient Descent (GD), to find . Following the discussion in 1-linear-regression.ipynb, consider minw,w0 L(w,w0) 1 2 ||y Xw w01n||22, which does not include the coefficients of the intercept in the design matrix X. Show that wL= XT (y Xw w01n), w0L= 1T (y Xw w01n).

Problem 5(20pts). (a) As shown in the lecture, if n>p, the estimated parameter of the regression model y=x+ is ^=(XX)1Xy, where XRnp and yRn are the data, and is a zero mean random variable with var()=2. Note that ^Rp is a random vector as it is function of the data. Show that var(^)=2(XX)1. (b) As discussed in the lecture, when np, we need to use an optimization algorithm, e.g., Gradient Descent (GD), to find ^. Following the discussion in "1-linear-regression.ipynb", consider minw,w0L(w,w0)21yXww01n22, which does not include the coefficients of the intercept in the design matrix X. Show that wLw0L=XT(yXww01n)=1T(yXww01n)