Suppose we are given a training dataset consisting of feature vectors

$x_1,$dots,$x_{N}in{R}^d$ and corresponding target values $y_1,$dots,$y_N$inR. The

vector $x_{i}in{R}^d$ can be written in more detail as

$x_i=\left[\begin{array}{c}{x}_{i1}\\ \text{v}\text{d}\text{o}\text{t}\text{s}\\ x_{id}\end{array}\right].$

We would like to find a vector

$=\left[\begin{array}{c}{}_0\\ {}_1\\ \text{v}\text{d}\text{o}\text{t}\text{s}\\ {}_d\end{array}\right]in{R}^{d+1}$

such that

$y_i$~~${}_0+{}_1{x}_{i1}+$cdots$+{}_d{x}_{id}$ for $i=1,$dots,$N.$

$($a$)$ For any given vector $in{R}^{d+1},$ the mean squared error can be written

as

$L()=\frac{1}{N}||x-y|{|}^2.$

What is the matrix $x$ and what is the vector $y?$ Explain why the

above formula computes the mean squared error.

$($b$)$ Let's compute $L^{\text{'}}()$ using the chain rule. The function $L$ can be

expressed as

$L()=g(h())$

for certain functions $g$ and $h.$ What are $g$ and $h?$ Write down for$-$

mulas for $h^{\text{'}}()$ and $g^{\text{'}}(u).$ What is $L^{\text{'}}()?$ What is gradL$()?$

$($c$)$ In single variable calculus, we learn to minimize functions by setting

the derivative equal to $0.$ In multivariable calculus, we can attempt

to minimize $L$ by setting gradL$()=0$ and solving for $.$ Write down

the equation gradL$()=0$ and simplify. Do we know of a technique

that can be used to solve for $?$

$($d$)$ Write Python code that uses the above approach to implement linear

regression from scratch using the California housing dataset. You can

use scikit$-$learn to load the dataset and split the dataset into training

and validation datasets. However, you should not use scikit$-$learn

to train your model. It's ok to use numpy to help with vector and

matrix operations.

$($e$)$ An alternative strategy to find a vector $in{R}^{d+1}$ which minimizes

$L()$ is to use the gradient descent algorithm. Write Python code

that trains a linear regression model from scratch using gradient de$-$

scent. You can use scikit$-$learn to load the California housing dataset

and to split it into training and validation datasets, but you should

not use scikit$-$learn to train your model. Make a plot $($called a "con$-$

vergence plot"$)$ that shows the value of $L()$ after each iteration of

gradient descent. Try out different learning rates. Which learning

rate leads to the fastest convergence? How many iterations of gradi$-$

ent descent do you need in order for the gradient descent iteration to

converge?