Using LASSO regression to build parsimonious model in R:

The purpose of this assignment is to use Least Absolute Shrinkage and Selection Operator (LASSO) to perform regularization and variable selection on a given model.

Depending on the size of the penalty term, LASSO shrinks less relevant predictors to (possibly) zero. Thus, it enables us to consider a more parsimonious model.

Please refer to questions and reference solutions with R codes (open the attached file for it) in which you will see how to use R for diabetes dataset (see the reference websites). Then use the NewYorkHousing.csv as attached we have used in assignment 1 and slightly modify the R codes you will be able to answer the following questions:

Load the lars package and the New York Housing dataset (as in the attached file for download).

Next, load the glmnet package that will be used to implement LASSO.

Save the MEDV as y and only use the first 12 variables (i.e., columns in New York Housing dataset) as x. While x is a set of independent variables, y is the dependent variable which is a quantitative measure of the median housing values. (Hints: can assign matrix format to the variable x).

Generate separate scatterplots with the line of best fit for all the predictors in x with y on the vertical axis.

Regress y on the predictors in x using OLS (Ordinary Least Square Regression). We will use this result as benchmark for comparison.

Use the glmnet function to plot the path of each of x's variable coefficients against the L1 norm of the beta vector. This graph indicates at which stage each coefficient shrinks to zero.

Use the cv.glmnet function to get the cross validation curve and the value of lambda that minimizes the mean cross validation error.

Using the minimum value of lambda from the previous exercise, get the estimated beta matrix.

To get a more parsimonious model we can use a higher value of lambda that is within one standard error of the minimum.

Use this value of lambda to get the beta coefficients. Note that more coefficients are now shrunk to zero.

t.