Linear Model Selection and Regularization

This programming assignment will use the Tidy Models platform. It will take a look at regularization models and hyperparameter tuning. These models contain a regularization term. This assignment will use parsnip for model fitting and recipes and workflows to perform the transformations, and tune and dials to tune the hyperparameters of the model.

You will be using the Hitters data set from the ISLR package. You wish to predict the baseball players Salary based on several different characteristics which are included in the data set.

Since you wish to predict Salary, then you need to remove any missing data from that column. Otherwise, you won't be able to run the models.

Set output as Hitters

library$($tidymodels$)$

library$($ISLR$2)$

$$

# Your code here

# Hitters $<-$

$$

# your code here

$$

# Hidden Tests

You will use the glmnet package to perform ridge regression. parsnip does not have a dedicated function to create a ridge regression model specification. You need to use linear$\_$reg$()$ and set mixture $=0$ to specify a ridge model. The mixture argument specifies the amount of different types of regularization, mixture $=0$ specifies only ridge regularization and mixture $=1$ specifies only lasso regularization.

Setting mixture to a value between $0$ and $1$ lets us use both. When using the glmnet engine you also need to set a penalty to be able to fit the model. You will set this value to $0$ for now, it is not the best value, but you will look at how to select the best value in a little bit.

ridge$\_$spec $<-$ linear$\_$reg$($mixture $=0,$ penalty $=0)\%>\%$

set$\_$mode$("$regression$")\%>\%$

set$\_$engine$("$glmnet$")$

Once the specification is created you can fit it to you data. You will use all the predictors. Use the fit function here.

ridge$\_$fit $<-$ fit$($ridge$\_$spec, Salary ~ $.,$ data $=$ Hitters$)$

The glmnet package will fit the model for all values of penalty at once, so you can now see see what the parameter estimate for the model is now that you have penalty $=0.$ You can use the tidy function to accomplish this specific task.

tidy$($ridge$\_$fit$)$

Let us instead see what the estimates would be if the penalty was $11498.$ Store your output to tidy$2.$ What do you notice?

# Your code here

$$

# tidy$2<-$

$$

$$

# your code here

$$

# Hidden Tests

Look below at the parameter estimates for penalty $=705.$ Store your output to tidy$3.$ Once again, use the tidy function to accomplish this task.

# Your code here

$$

# tidy$3<-$

$$

$$

# your code here

$$

# Hidden Tests

You can visualize how the magnitude of the coefficients are being regularized towards zero as the penalty goes up$.$ Use the autoplot$()$ function to accomplish this task. Output variable here is ridge$\_$fit. Your image should look like this:

ridge$\_$fit $<-\%>\%$ autoplot$()$

Prediction is done like normal, if you use predict$()$ by itself, then penalty $=0$ as you set in the model specification is used.

predict$($ridge$\_$fit, new$\_$data $=$ Hitters$)$

But you can also get predictions for other values of penalty by specifying it in predict$().$ Test with a value of $500.$ Store your output to predict$\_500$

# Your code here

$$

# predict$\_500<-$

$$

# your code here

$$

# Hidden Tests

You saw how we can fit a ridge model and make predictions for different values of penalty. But it would be great if you could find the "best" value of the penalty. This is something you can use hyperparameter tuning for. Hyperparameter tuning is in its simplest form a way of fitting many models with different sets of hyperparameters trying to find one that performs "best".

The complexity in hyperparameter tuning can come from how you try different models. You will keep it simple for this lab and only look at grid search, only looking at evenly spaced parameter values. This is a fine enough approach if you have one or two tunable parameters but can become computationally infeasible.

See the chapter on iterative search from Tidy Modeling with R for more information.

You begin like normal by setting up a validation split $($testing and training set$).$ A K$-$fold cross$-$validation data set is created on the training data set with $10$ folds.

Hitters$\_$split $<-$ initial$\_$split$($Hitters$,$ strata $=$ "Salary"$)$

Hitters$\_$train $<-$ training$($Hitters$\_$split$)$

Hitters$\_$test $<-$ testing$($Hitters$\_$split$)$

Hitters$\_$fold $<-$ vfold$\_$cv$($Hitters$\_$train, v $=10)$

You can use the tune$\_$grid$()$ function to perform hyperparameter tuning using a grid search. tune$\_$grid$()$ needs $3$ different things;

a workflow object containing the model and preprocessor,

a rset object containing the resamples the workflow should be fitted within, and

a tibble containing the parameter values to be evaluated.

Optionally a metric set of performance metrics can be supplied for evaluation. If you don't set one then a default set of performance metrics is used.

You already have a resample object created in Hitters$\_$fold. Now you should create the workflow specification next.

You just used the data set as is when you fit the model earlier. However, ridge regression is scale sensitive so you need to