Please answer all the question with a clear explanation, thank you for your work!
Consider the algorithm for building a CART model in the case of regression. Following and ex- panding on the notation from class, suppose that our current tree, denoted by Told, has Told = M terminal nodes/buckets. For each bucket m= 1,..., M, let: 1. N, denote the number of observations in bucket m, 2. Qm(Told) denote the value of the impurity function at bucket m , and 3. Rm denote the region in the feature space corresponding to bucket m . Also let N be the overall total number of observations. Recall that, in the case of regression we have that: Qm(Told) (y: - .)?, it,CR where nn = Nome Liz:CrmY; is the mean response in bucket m. Then the total impurity cost of the tree Told is defined as: M Cimp(Told) = N,Qm(Told). m=1 Consider a potential split at the final bucket M (we're using M just for case of notation), which results in a new tree Tnew. This new tree has Thewl = M +1 terminal nodes/buckets, and for this new tree we let 1. Non denote the number of observations in bucket m, 2. (Tnew) denote the value of the impurity function at bucket m , and 3. Rm denote the region in the feature space corresponding to bucket m The total impurity cost of the tree Thew is defined analogously as: M1 Cimp(Tow) = [Nm(Thew). T=1 Please answer the following: 72 a) (10 points) Let A Cimp (Told) - Cimp (Tnew) be the absolute decrease in total impurity resulting from the split. Derive a formula for A that can be computed locally at the bucket M, in other words it should only depend on the data points that fall in region Rm in the original tree Told. (Hint: we've discussed this concept in class, this question is asking for a more formal argument. You may assume that the two new buckets in Thew resulting from the split are labeled as buckets M and M +1 in Thew.) b) (10 points) Show that A > 0, hence splitting always reduces the total impurity cost. (Hint: you can use the fact that, given a sequence of real numbers 21, 22, ..., Zn, the mean z :1 zi is the minimizer of the function RSS(z) = ?-1(21 - 2)2) c) (10 points) Let Rld be the training set R2 value for the model defined by Told, and likewise let Rew be the training set R2 value for the model defined by Tnew. Let SST = E(Yi y)2 be the total sum of squared errors, where y=-1 Y; is the overall mean. For a given value of the complexity parameter (cp) a > 0, recall the modified cost function that is relevant in the pruning step: Ca(T) = Cimp(T) + QSST. T|| Show that Ca(Tnew) a. (Hence the choice of retaining a split if the increase in R2 is at least a is equivalent to retaining a split if the modified cost function is smaller after the split.) Consider the algorithm for building a CART model in the case of regression. Following and ex- panding on the notation from class, suppose that our current tree, denoted by Told, has Told = M terminal nodes/buckets. For each bucket m= 1,..., M, let: 1. N, denote the number of observations in bucket m, 2. Qm(Told) denote the value of the impurity function at bucket m , and 3. Rm denote the region in the feature space corresponding to bucket m . Also let N be the overall total number of observations. Recall that, in the case of regression we have that: Qm(Told) (y: - .)?, it,CR where nn = Nome Liz:CrmY; is the mean response in bucket m. Then the total impurity cost of the tree Told is defined as: M Cimp(Told) = N,Qm(Told). m=1 Consider a potential split at the final bucket M (we're using M just for case of notation), which results in a new tree Tnew. This new tree has Thewl = M +1 terminal nodes/buckets, and for this new tree we let 1. Non denote the number of observations in bucket m, 2. (Tnew) denote the value of the impurity function at bucket m , and 3. Rm denote the region in the feature space corresponding to bucket m The total impurity cost of the tree Thew is defined analogously as: M1 Cimp(Tow) = [Nm(Thew). T=1 Please answer the following: 72 a) (10 points) Let A Cimp (Told) - Cimp (Tnew) be the absolute decrease in total impurity resulting from the split. Derive a formula for A that can be computed locally at the bucket M, in other words it should only depend on the data points that fall in region Rm in the original tree Told. (Hint: we've discussed this concept in class, this question is asking for a more formal argument. You may assume that the two new buckets in Thew resulting from the split are labeled as buckets M and M +1 in Thew.) b) (10 points) Show that A > 0, hence splitting always reduces the total impurity cost. (Hint: you can use the fact that, given a sequence of real numbers 21, 22, ..., Zn, the mean z :1 zi is the minimizer of the function RSS(z) = ?-1(21 - 2)2) c) (10 points) Let Rld be the training set R2 value for the model defined by Told, and likewise let Rew be the training set R2 value for the model defined by Tnew. Let SST = E(Yi y)2 be the total sum of squared errors, where y=-1 Y; is the overall mean. For a given value of the complexity parameter (cp) a > 0, recall the modified cost function that is relevant in the pruning step: Ca(T) = Cimp(T) + QSST. T|| Show that Ca(Tnew) a. (Hence the choice of retaining a split if the increase in R2 is at least a is equivalent to retaining a split if the modified cost function is smaller after the split.)
