Derivation of Theoretical Linear Least Squares Regression | 2i2 points [graded] Normally, we should be thinking of linear regression being performed on a data set {(2.}, 29%)};1: which we think of as a deterministic collection of points in the Euclidean space. It is helpful to also consider an idealized scenario, where we assume that X and Y are random variables that follow some joint probability distribution and they have finite first and second moments. In this problem, we will derive the solution to the theoretical linear regression problem. Assume Var (X) 75 D. The theoretical linear [least squares) regression of Y on X prescribes that we find a pair of real numbers a and b that minimize ]E [{Y a E3102], over all possible choices of the pair (a, b]. To do so. we will use a classical calculus technique. Let f (a, b) = E [(Y or. 5102]. and now we solve for the critical points where the gradient is zero. Hint: Here, assume you can switch expectation and differentiation with respect to a. and b. That is, aaE [( ' )] 2 E [Ba {' ' 'H' Use X and Y for random variables X and Y. The partial derivatives are: 3&3" = 4'.[ 2*[Y-a-b*X} v ] w:_ i Derivation of Theoretical Linear Least Squares Regression ll Lil point [graded] Setting these equal to zero and isolating terms with a and b to one side, we obtain a system of linear equations Ew1=a+Emw smyyamma+smb Multiplying the first equation by E [X] and subtracting from the second equation gives . r 2 _ _ _ _Cov[X,Y) (E[12]IE:[X])b]E[XY] EleEll'] > b_Var(X)' Plugging this value back into the first equation to solve for (1 gives _ Cov(X,Y). Var{X) ' a=Em mm. We now compute the Hessian H = (fat! fab ) fa: flab to make sure that this pair (a, .5] critical point is a local minimum. The determinant of H at this value (a, b) is O Var (X) 4Var(X) Theoretical Linear Regression Visualized l 1 point possible lgra dodl Consider again the setting or theoretical linear regression, as in the previous problems on this page. Let X, Y be random variables such that Var (X) 7': D. Assume ]E. [X] and ]F. [Y] are both zero. Let a, b be solutions that minimze the squared error CV{X'Y)]EL[X1, b Cov(.X,Y) \"W\" mm \"W which gives the bestfitting line E [Y|X = e] \"a: a i be. Assume that the line '5; "- a + be looks like: 20 15 10 In particular, a = D due to our simplifying assumptions. If Y" is a different random variable such that ]E [Y'] = 0, Cat: (X, Y') 3 COM! (X, Y), which of the following choices best illustrates, via a newline drawn in red, the theoretical linear regression of the pair X, Y\"? \fTheoretical Linear Regression Visualized II 1 point possible (graded) Now consider the same setting as in the previous problem, except we drop the assumption IE [Y ] = 0, and we now assume EX >O. Again, let a, b be solutions that minimze the squared error, so that the line y = a + ba looks like: 10 10 If Y is a different random variable such that Cov (X, Y") > Cov (X, Y) and E [Y] > E [Y"'], which of the following choices best illustrates, via a new line drawn in red, the theoretical linear regression of the pair X, Y'? O 10 10