Predicting housing median prices.

The file BostonHousing.xls contains information on over 500 census tracts in Boston, where for each tract 14 variables are recorded. The last column (CAT.MEDV) was derived from MEDV, such that it obtains the value 1 if MEDV>30 and 0 otherwise. Consider the goal of predicting the median value (MEDV) of a tract, given the information in the first 13 columns. Partition the data into training (60%) and validation (40%) sets. (For interpretation of the column names in BostonHousing.xls, please make reference to Table 2.2 on page 27 of the textbook)

a) Perform a k-nearest neighbors prediction with all 13 predictors (the CAT.MEDV column is the outcome or decision variable), trying values of k from 1 to 10. Make sure to normalize the data (click normalize input data"). What is the best k chosen? What does it mean? (10 points)

b) Why is the validation data error overly optimistic compared to the error rate when applying this kNNpredictor to new data? (10 points)