The variance of the regression prediction and variance of the least squares estimators tend to be larger when the regression model is overfitted. Show the following results for an over-fitted regression model:

(a) Let x1, x2, ··· , xk be the regressors and b0, b1, ··· , bk be the estimates of the regression model, if (k + 1)-th regressor xk+1 is added into the model and the estimates are denoted by b∗ 0, b∗ 1, ··· , b∗ k, b∗ k+1 then Var(b∗ i ) ≥ Var(bi) for i = 0, 1, 2, ··· , k.

(b) The variance of fitted value based on an over-fitted model is larger. i.e., denote ˆy1 = Pk i=0 bixi and ˆy2 = Pk+1 i=0 b ∗ i xi , then Var(ˆy1) ≤ Var(ˆy2).