Suppose we are interested in solving a linear system Ax = b by the method of least squares when the coefficient matrix A has linearly dependent columns. Let Kx = f, where K = ATCA, f = ATCb, be the corresponding normal equations.
(a) Prove that f ∈ mg K, and so the normal equations have a solution. Use Exercise 3.4.31.
(b) Prove that any solution to the normal equations minimizes the least squares error, and hence qualifies as a least squares solution to the original system.
(c) Explain why the least squares solution is not unique.