1. What is the overall proportion of the Web Robot in the original dataset?
2. Partition the data into training (60%) and validation (40%) (set the random seed to 12345). Using the same training and validation sets, build a classification tree with the training data, report the tree graph. Calculate and report the Gini index for the split of the top node. Show your work on how to calculate the Gini index for left and right child (branch) and the overall Gini for the split. Apply the model on the validation set, report the classification table, the overall accuracy and misclassification error rate on the validation data.
3. Suppose we now have new observation with some missing values. The only information available to us is as follows: Depth=1, Breadth=1, ImagePages=0.75. Following the structure of classification tree you obtained in part (c), is it possible to make a predictor whether this web access is from Robot or not? If yes, please classify the new observation as Robot_Yes or Robot_No. Provide a brief explanation.
4. With the same training and validation sets, use the random forest method to build a classification tree with the training data, generate the variable importance plot and report the top three most important variables. Report the classification table (frequency counts), the overall accuracy rate, misclassification error rate on the validation dataset.
5. Compare the models you obtained in parts (b) and (d), which model is the best with respect to the misclassification error rate performance on the validation dataset.