Question

For leave-one-out cross validation (or equivalently c-fold cross validation with c = n) the cross validation error Eva for ridge regression actually has a closed- form solution Eval = yi - yi IM= n - [ H(v)]i. (1) where y; is the prediction of y; when the model is learned from all n data points (no data point is left out), y the regularization parameter and [H(y)] is element (i, i) of the matrix H(y) = X(X X + 1)-iXT. In this problem, we will let x, denote the entire ith row of X. (a) Let X_; denote the matrix X where row i is removed, and y_; is the column vector y with element i removed. Show that XIX_ = XX - xx], XI,y_ = Xy - Xiyi, and [H(Y)]a = x] (XX + 71) 'x: (b) Using the results from (a) and a special case of the matrix inversion lemma (A - VV)-1 = A-1 A vvTA-1 1 - VIA TV' show that 1 3_, = B+ 1- H(7)] u (XX+yl) 'x;(yi - yi). Here _; are the parameters learned from all data points except i, and 3 the parameters learned from all data. Hint: Start from the ridge regression expression for B_, as a function of X_i, y-; and . (c) Use your result from (b) to derive eq. (1), starting from (d) Describe (in a few sentences) what eq. (1) can be used for in practice