Problem Statement:

The goal of Part I of the task is to use raw textual data in language models for recommendation based application.

The goal of Part II of task is to implement comprehensive preprocessing steps for a given dataset, enhancing the quality and relevance of the textual information. The preprocessed text is then transformed into a feature$-$rich representation using a chosen vectorization method for further use in the application to perform similarity analysis.

Part I

Sentence comparison using N$-$gram: $(3$ Marks$)$

Let a search engine powered by language model recommend which of the below sentences are most relevant w$.$r$.$t to given training corpus. Design a probabilistic language model to compare below test sentences for recommendation using Trigram. Use all the instances in the dataset as a training corpus.

Test Sentence : Affection.

Part II

Perform the below sequential tasks on the given dataset.

i$)$ Text Preprocessing: $(2$ Marks$)$

Tokenization

Lowercasing

Stop Words Removal

Stemming

Lemmatization

ii$)$ Feature Extraction: $(2$ Marks$)$

Use the pre$-$processed data from previous step and implement the below vectorization methods to extract features.

Word Embedding using TD$-$IDF

iii$)$ Similarity Analysis: $(3$ Marks$)$

Use the vectorized representation from previous step and implement a method to identify and print the names of top two similar words that exhibit significant similarity. Justify your choice of similarity metric and feature design. Visualize a subset of vector embedding in $2$D semantic space suitable for this use case. HINT: $($Use PCA for Dimensionality reduction