1 Logistic Regression with Newton's Method Consider sample points x,.x2. . . . , X" E R" and associated values yl+N2, design matrix X-(X! . . . Kol and an n-vector y-[y| . . . yn]. If we add (2-regularization to logistic regression, the cost function is y" E {0, l}, an n x d IF where si-s(Xi . w), s(y)-1/(1 + e-y), and > 0 is the regularization parameter. As in lecture. the vector s - [s1... S,] is a useful shorthand In this problem, you will use Newton's method to minimize this cost function on the four-point, two-dimensional training set 0 0 You may want to draw these points on paper to see what they look like. The y-vector implies that the first two sample points are in class 1, and the last two are in class 0 These sample points cannot be separated by a linear decision boundary that passes through the origin. As described in lecture, append a 1 to each X, vector and use a weight vector w eR3 whose last component is the bias term (the term we call a in lecture) 1. Derive the gradient of the cost function J(w). Your final answer should be a simple matrix- vector expression. While you may derive the matrix-vector form by first deriving the com ponents of the gradient vector, do NOT write your answer in terms of these individual com- ponents 2. Derive the Hessian of J(w). Again, your answer should be a simple matrix-vector expression 3. State the update equation for one iteration of Newton's method for this problenm 4. we are given a regularization parameterof -0.07 and a starting point of w(0)-1-2 1 0 For the following four parts, you need only state the final solution. Thus you may derive the solution by hand or implement Newton's algorithm and report the final result. If you do the latter, you do not need to submit code for this part. (a) State the value of s (the value of s before any iterations) (b) State the value of w (the value of w after one iteration) (c) State the value of s). (d) State the value of w2) (the value of w after two iterations) 1 Logistic Regression with Newton's Method Consider sample points x,.x2. . . . , X" E R" and associated values yl+N2, design matrix X-(X! . . . Kol and an n-vector y-[y| . . . yn]. If we add (2-regularization to logistic regression, the cost function is y" E {0, l}, an n x d IF where si-s(Xi . w), s(y)-1/(1 + e-y), and > 0 is the regularization parameter. As in lecture. the vector s - [s1... S,] is a useful shorthand In this problem, you will use Newton's method to minimize this cost function on the four-point, two-dimensional training set 0 0 You may want to draw these points on paper to see what they look like. The y-vector implies that the first two sample points are in class 1, and the last two are in class 0 These sample points cannot be separated by a linear decision boundary that passes through the origin. As described in lecture, append a 1 to each X, vector and use a weight vector w eR3 whose last component is the bias term (the term we call a in lecture) 1. Derive the gradient of the cost function J(w). Your final answer should be a simple matrix- vector expression. While you may derive the matrix-vector form by first deriving the com ponents of the gradient vector, do NOT write your answer in terms of these individual com- ponents 2. Derive the Hessian of J(w). Again, your answer should be a simple matrix-vector expression 3. State the update equation for one iteration of Newton's method for this problenm 4. we are given a regularization parameterof -0.07 and a starting point of w(0)-1-2 1 0 For the following four parts, you need only state the final solution. Thus you may derive the solution by hand or implement Newton's algorithm and report the final result. If you do the latter, you do not need to submit code for this part. (a) State the value of s (the value of s before any iterations) (b) State the value of w (the value of w after one iteration) (c) State the value of s). (d) State the value of w2) (the value of w after two iterations)
