Problem $1$ TF$-$IDF

Implement TF$-$IDF using using Python, Numpy, Pandas and whatever text cleaning library required.

The tf$$idf is the product of two statistics, term frequency and inverse document frequency. There are various ways for determining the exact values of both statistics, you can use the following formulas.

Term Frequency

tft$,$d$=$log$10($count$($t$,$d$)+1)$

tft$,$d is the frequency of the word t in the document d

Inverse Document Frequency

idft$=$log$10($Ndft$)$

N is the total number of documents

dft is the number of documents in which term t occurs

TF$-$IDF

tf$-$idft,d$=$tft$,$d$\backslash$times idft

What is expected?

Your implementation should include the following two functions:

compute$\_$tfidf$\_$weights$($train$\_$docs$)$

word$\_$tfidf$\_$vector$($word$,$ tf$\_$df$,$ idf$\_$df$)$

$[]$

def compute$\_$tfidf$\_$weights$($train$\_$docs$)$:

$\text{'}\text{'}\text{'}$

Input arguments:

train$\_$docs : list of documents, i$.$e$.,$ strings

Output arguments:

docs$\_$tf : tf as a DataFrame

docs$\_$idf : idf as a DataFrame

Note: the use of Pandas DataFrame is not mandatory

$\text{'}\text{'}\text{'}$

docs$\_$tf $=$ None

docs$\_$idf $=$ None

return docs$\_$tf$,$ docs$\_$idf

def word$\_$tfidf$\_$vector$($word$,$ tf$\_$df$,$ idf$\_$df$)$:

$\text{'}\text{'}\text{'}$

Input arguments:

word : a query string

tf$\_$tf : tf as a DataFrame

tf$\_$idf : idf as a DataFrame

Output arguments:

tf$\_$idf$\_$value : a numpy array of dimension $1$xN

$\text{'}\text{'}\text{'}$

tf$\_$idf$\_$value $=$ None

return tf$\_$idf$\_$value

Problem $2$ Word embedding as features for classification

Task

Implement a sentiment classifier based on Twitter data to analyse the sentiments of COVID$-19$ tweets.

Train and test multiple classification model using necessary libraries with the features being sentence embeddings of tweets.

Report the accuracy and F$1$ score $($micro$-$ and macro$-$averaged$)$ for multiple classifier and discuss the differences.

Dataset

The dataset have been provided in the first code trunk with the assignment. You are required to use the original tweet text for this classification task.

Tweet representation

After necessary pre$-$processing of the tweets, convert the words into their embeddings, then take the mean of all the word vectors in a tweet to end up with a single vector representing each tweet. The tweet vector is then used for sentiment classification.

In the process of finding the embeddings for each word, you can ignore out$-$of$-$vocabulary words.

Classifier choice

You are required to implement the following TWO classifiers:

One tradition classification model $($not a neural network based model$)$

One classifier based on any neural network based model.

You can use PyTorch$/$TensorFlow$/$scikit$-$learn to implement your classifier. However, you are free to develop a classifier from scratch.

Your answer must include the following:

Code for data loading, data pre$-$processing, training, and testing of the models.

A discussion on the comparison between the classifiers based on classifier accuracy and F$1$ score