1. Bias-variance tradeoff. Suppose I want to identify the location of a star, which lives at coordinates defined by XER. Every day I go to the telescope, and I receive a new measurement yi = x + zi, where zi ~ N(0,1) is the noise in each measurement. After m days, I receive m measurements, 41, ..., Ym ER. (a) Denote XMLE as the maximum likelihood estimate of r. Compute XMLE in terms of yi and m. Compute also the bias and variance of XMLE. Describe the behavior of the bias and variance as m + too. (b) A colleague walks in the room and scoffs at my experiment. I already know where this star is!" the colleague exclaims, and gives me a new set of measurements ER. You can just cancel your experiment now!" Trouble is, I know the colleague is full of hot air, so while this is valuable information, I'm not willing to take it without any verification. Instead, I estimate x by solving a linear regression problem m minimize (11 2)2 + (x ) 1 m i=1 for some p > 0. Denote XMAP as the solution to this linear regression problem. Write out XMAP in terms of Yi, , p, and m. Compute also the bias and variance of XMAP, its behavior as m + too, and how it compares to the bias and variance of the MLE estimator. (c) Show that for any estimator , the expected squared error E[(x )2] = B2 + V where B = x E[a] the estimator bias and V = E[(El] )2] the estimator variance. (d) In terms of constants A = x and m, what value of p minimizes the squared error of the estimator XMAP? Hint: if a function is not convex in a, it may be convex in where = f(a) is a 1-1 function of a. 1. Bias-variance tradeoff. Suppose I want to identify the location of a star, which lives at coordinates defined by XER. Every day I go to the telescope, and I receive a new measurement yi = x + zi, where zi ~ N(0,1) is the noise in each measurement. After m days, I receive m measurements, 41, ..., Ym ER. (a) Denote XMLE as the maximum likelihood estimate of r. Compute XMLE in terms of yi and m. Compute also the bias and variance of XMLE. Describe the behavior of the bias and variance as m + too. (b) A colleague walks in the room and scoffs at my experiment. I already know where this star is!" the colleague exclaims, and gives me a new set of measurements ER. You can just cancel your experiment now!" Trouble is, I know the colleague is full of hot air, so while this is valuable information, I'm not willing to take it without any verification. Instead, I estimate x by solving a linear regression problem m minimize (11 2)2 + (x ) 1 m i=1 for some p > 0. Denote XMAP as the solution to this linear regression problem. Write out XMAP in terms of Yi, , p, and m. Compute also the bias and variance of XMAP, its behavior as m + too, and how it compares to the bias and variance of the MLE estimator. (c) Show that for any estimator , the expected squared error E[(x )2] = B2 + V where B = x E[a] the estimator bias and V = E[(El] )2] the estimator variance. (d) In terms of constants A = x and m, what value of p minimizes the squared error of the estimator XMAP? Hint: if a function is not convex in a, it may be convex in where = f(a) is a 1-1 function of a
