Generate a training dataset with 50 observations using the following code: x1 <- runif(50) x2 <- runif(50) eps <- rnorm(50,mean=0,sd=0.25) y <- x1+x2+eps df <- data.frame(y,x1,x2) Generate another test dataset with 30 observations using the same setup. Use the training data to construct a linear model with X1 and X2 no interaction terms or higher-order polynomial terms. Calculate the MSE on the test set and call this MSE0.

(b) Recall that an (overall) F-test tests whether any of the features are significant. Using the test MSE as our statistic, devise a permutation-test-equivalent to the overall F-test. Carry out the test with 1000 permutations. Can you reject the null hypothesis that none of the predictors are significant using your test?

(c) As we know, we can examine whether an individual feature is significant by looking at its corresponding t-test. Again using test MSE as the statistic of interest, devise a permutation-test-equivalent to the individual t-test. Carry out the test with 1000 permutations to test whether X2 is significant. Can you reject the null hypothesis that 2 = 0 using your test?

(d) In this case, we have only two features so an individual t-test is equivalent to a partial F-test. Lets scale things up a bit. Using the same general procedure and model as above, 2 create a training set with 500 observations on 10 features. Also create a test set with 50 observations.

(e) Using the data from part(d) and using test MSE as the statistic of interest, devise a permutation-test-equivalent to the partial F-test that will evaluate whether any of the features X8, X9 or X10 are significant. Carry out the test with 1000 permutations. Can you reject the null hypothesis?