$2$ Fitting a Linear Model

Problem $2$: Write the following functions.

A function that takes as input a data tensor $x$ and vector of weights $w_{?},$ and returns a tensor of predictions $y_{{?}^{?}}(pred)$ based on a linear model, $w_{*}{x}_{?}$ for each data point.

A loss function that takes as input a tensor of actual $y-$values, $y_{?},$ and a vector of predicted $y-$values, $y_{{?}^{?}}(pred),$ and returns $($as a result of tensor operations$)$ the average squared error between them. $($Average in this case being over the number of data points in the tensors.$)$

A function that may be useful: torch.matmul.

Problem $3$: Given a training data set $(x^{\text{t}\text{r}\text{a}\text{i}\text{n}},y_{{?}^{?}}(train)),$ and a testing data set $(x^{\text{t}\text{e}\text{s}\text{t}},y_{{?}^{?}}(test)),$ and a stepsize $>0,$ write code that calculates a linear model of best fit on the training data using a stepsize of $.$ Keep track of the loss on the training set and the loss on the testing set over the course of training, to plot at the end of training. What condition should you put to end your training loop?

Problem $4$: For a training set of size $N=1000$ and a test set of size $N=200,$ consider fitting a linear model at various stepsizes $.$

What range of $$ lead to convergence, what don't? How can you tell that an $$ is too large?

For an $$ with convergence, plot the loss on the training data and the loss on the testing data as a function of training time. How do the training and testing error compare over time? Is the model learning?

For a good $,$ trained to convergence $-$ does the resulting model generalize to the testing data well?

How does the fit model compare to the 'actual' model?

Bonus: Show that there are multiple models that produce the same output, and therefore there is no 'best' model. Does this contradict there being a 'true' formula, as indicated above?

$2$

Problem $5$: In this problem, we want to look at the effect of the amount of data. For $N=1000,500,250,100,50,$ consider training a model on training sets of these sizes $($keeping the test set fixed at size $N=200).$ Plot the training and testing loss for each experiment over the course of training. Analyze and describe your results.

Bonus: How does the optimal $$ seem to depend on the number of training data points? Be thorough.