In this project, we will be developing a basic neural network from the ground up to classify various types of fashion items. The primary objective of this project is to gain a comprehensive understanding of neural network architecture, including its theory and implementation details.

Categories in the dataset:

$0$: T$-$shirt$/$top

$1$: Trouser

$2$: Pullover

$3$: Dress

$4$: Coat

$5$: Sandal

$6$: Shirt

$7$: Sneaker

$8$: Bag

$9$: Ankle boot

# Notice that you don't need any other packages for this mid$-$term

import numpy as np

import pandas as pd

import random

from matplotlib import pyplot as plt

random.seed$(42)$ # NEVER change this line; this is for grading

# Reading the dataset

data $=$ pd$.$read$\_$csv$(\text{'}./$fashion$\_$data.csv$\text{'})$

# The data pre$-$processing is done for you. Please do NOT edit the cell

# However, you should understand what these codes are doing

data $=$ np$.$array$($data$)$

m$,$ n $=$ data.shape

np$.$random.shuffle$($data$)$ # shuffle before splitting into dev and training sets

data$\_$dev $=$ data$[0$:$400].$T

Y$\_$dev $=$ data$\_$dev$[-1]$

X$\_$dev $=$ data$\_$dev$[0$:n$-1]$

X$\_$dev $=$ X$\_$dev $/255.$

data$\_$train $=$ data$[400$:m$].$T

Y$\_$train $=$ data$\_$train$[-1]$

X$\_$train $=$ data$\_$train$[0$:n$-1]$

X$\_$train $=$ X$\_$train $/255.$

$\_,$m$\_$train $=$ X$\_$train.shape

# define a global variable specifying the number of hidden neurons after the first layer

# not the best practice, but we will do it for this mid$-$term project

num$\_$hidden$\_$neurons $=20$

# Initialize the parameters in the neural network

# Based on the figure above, we need the weight and bias matrices.

# W$1,$ b$1$ are the matrices for the first layer

# W$2,$ b$2$ are the matrices for the second layer

# You should think about the sizes of the matrices

# then initialize elements in the matrix to be random numbers between $-0\mathrm{.}5$ to $+0\mathrm{.}5$

def init$\_$params$()$:

W$1=$ # Your code here

b$1=$ # Your code here

W$2=$ # Your code here

b$2=$ # Your code here

return W$1,$ b$1,$ W$2,$ b$2$

# As a starting point, you only need a ReLu function, its derivative, and the softmax function

def ReLU$($Z$)$:

# Your code here

def ReLU$\_$deriv$($Z$)$:

# Your code here

def softmax$($Z$)$:

# Your code here

return A

# In the forward propagation function, X is the inputs $($the image in vector form$),$ and we pass all the weights and biases

def forward$\_$prop$($W$1,$ b$1,$ W$2,$ b$2,$ X$)$:

Z$1=$ # Your code here

A$1=$ # Your code here

Z$2=$ # Your code here

A$2=$ # Your code here

return Z$1,$ A$1,$ Z$2,$ A$2$

# This one hot function is to convert a numeric number into a one$-$hot vector

def one$\_$hot$($Y$)$:

# Your code here

return one$\_$hot$\_$Y

# Now performing the backward propagation

# Each function is only one line, but lots of Calculus behind

def backward$\_$prop$($Z$1,$ A$1,$ Z$2,$ A$2,$ W$1,$ W$2,$ X$,$ Y$)$:

one$\_$hot$\_$Y $=$ one$\_$hot$($Y$)$

dZ$2=$ # Your code here

dW$2=$ # Your code here

db$2=$ # Your code here

dZ$1=$ # Your code here

dW$1=$ # Your code here

db$1=$ # Your code here

return dW$1,$ db$1,$ dW$2,$ db$2$

# Finally, we are ready to update the parameters

def update$\_$params$($W$1,$ b$1,$ W$2,$ b$2,$ dW$1,$ db$1,$ dW$2,$ db$2,$ alpha$)$:

W$1=$ # Your code here

b$1=$ # Your code here

W$2=$ # Your code here

b$2=$ # Your code here

return W$1,$ b$1,$ W$2,$ b$2$

# Implement the helper function. We need to convert the softmax output into a numeric label

# This is done through get$\_$predictions function

def get$\_$predictions$($A$2)$:

# Your code here

# We also want to have a simple function to compute the accuracy. Notice that "predictions" and $"$Y$"$ are the same shape

def get$\_$accuracy$($predictions$,$ Y$)$:

return # Your code here

# Finally, we are ready to implement gradient descent

def gradient$\_$descent$($X$,$ Y$,$ alpha, iterations$)$:

W$1,$ b$1,$ W$2,$ b$2=$ # Your code here $-$ using the function you have implemented

for i in range$($iterations$)$:

Z$1,$ A$1,$ Z$2,$ A$2=$ # Your code here $-$ using the function you have implemented

dW$1,$ db$1,$ dW$2,$ db$2=$ # Your code here $-$ using the function you have implemented

W$1,$ b$1,$ W$2,$ b$2=$ # Your code here $-$ using the function you have implemented

if i $\%10==0$:

print$("$Iteration: $",$ i$)$

predictions $=$ get$\_$predictions$($A$2)$

print$($get$\_$accuracy$($predictions$,$ Y$))$

return W$1,$ b$1,$ W$2,$ b$2$

#Validate set performance

def make$\_$predictions$($X$,$ W$1,$ b$1,$ W$2,$ b$2)$:

$\_,\_,\_,$ A$2=$ forward$\_$prop$($W$1,$ b$1,$ W$2,$ b$2,$ X$)$

predictions $=$ get$\_$predictions$($A$2)$

return predictions

dev$\_$predictions $=$ make$\_$predictions$($X$\_$dev, W$1,$ b$1,$ W$2,$ b$2)$

get$\_$accuracy$($dev$\_$predictions, Y$\_$dev$)$

Part $2$: Error Analysis and Performance Improvements

You now will try to improve the model performance through, for example, different activation functions, learning rate cahnges, expanding the network complexity, regularization, and dropouts. Implement these ideas for improvement and compare to the base model you completed in part $1.$ The ideas may not improve the model, they may improve, give reasons why you did or did not succeed in makeing a better model. You must implement at least three different models.