Part I

Sentence completion using N$-$gram: $(3$ Marks$)$

Recommend the top $3$ words to complete the given sentence using N$-$gram language model. The goal is to demonstrate the relevance of recommended words based on the occurrence of Bigram within the corpus. Use all the instances in the dataset as a training corpus.

Test Sentence: "how could $\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_."$

#Load necessary packages

import pandas as pd

import nltk

nltk$.$download$(\text{'}$punkt$\text{'})$

from nltk import bigrams

from collections import defaultdict

import random

# Load the movie dialog dataset

df $=$ pd$.$read$\_$csv$($

$"/$content$/$movie$-$dialog$-$corpus$/$movie$\_$lines.tsv$",$

encoding$=$'utf$-8-$sig',

sep$=\text{'}\backslash$t$\text{'},$

on$\_$bad$\_$lines$=$"skip",

header $=$ None,

names $=[\text{'}$lineID$\text{'},$ 'charID', 'movieID', 'charName', 'text'$],$

index$\_$col$=[\text{'}$lineID$\text{'}]$

$)$

# Create a placeholder for the bigram model

bigram$\_$model $=$ defaultdict$($lambda: defaultdict$($lambda: $0))$

# Count frequency of co$-$occurrence for bigrams

for sentence in df$[\text{'}$text$\text{'}].$dropna$()$:

if isinstance$($sentence$,$ str$)$:

tokens $=$ nltk$.$word$\_$tokenize$($sentence$)$

for w$1,$ w$2$ in bigrams$($tokens$,$ pad$\_$right$=$True, pad$\_$left$=$True$)$:

bigram$\_$model$[$w$1][$w$2]+=1$

# Let's transform the counts to probabilities for bigrams

for w$1$ in bigram$\_$model:

total$\_$count $=$ float$($sum$($bigram$\_$model$[$w$1].$values$()))$

for w$2$ in bigram$\_$model$[$w$1]$:

bigram$\_$model$[$w$1][$w$2]/=$ total$\_$count

# Generate three sentences

for $\_$ in range$(3)$:

# starting word

text $=["$how$",$ "could"$]$

sentence$\_$finished $=$ False

while not sentence$\_$finished:

# select a random probability threshold

r $=$ random.random$()$

accumulator $=\mathrm{.}0$

if isinstance$($text$[-1],$ str$)$:

tokens $=$ nltk$.$word$\_$tokenize$($text$[-1])$

for word in bigram$\_$model$[$tokens$[-1]].$keys$()$:

accumulator $+=$ bigram$\_$model$[$tokens$[-1]][$word$]$

# select words that are above the probability threshold

if accumulator $>=$ r:

text.append$($word$)$

break

if text$[-1]==\text{'}.\text{'}$:

sentence$\_$finished $=$ True

generated$\_$sentence $=\text{'}\text{'}.$join$([$t for t in text if t$])$

print$($generated$\_$sentence$)$

Problem Statement:

The goal of Part I of the task is to use raw textual data in language models for recommendation based application.

The goal of Part II of task is to implement comprehensive preprocessing steps for a given dataset, enhancing the quality and relevance of the textual information. The preprocessed text is then transformed into a feature$-$rich representation using a chosen vectorization method for further use in the application to perform similarity analysis.

Part I

Sentence completion using N$-$gram: $(3$ Marks$)$

Recommend the top $3$ words to complete the given sentence using N$-$gram language model. The goal is to demonstrate the relevance of recommended words based on the occurrence of Bigram within the corpus. Use all the instances in the dataset as a training corpus.

Test Sentence: "how could $\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_."$

Part II

Perform the below sequential tasks on the given dataset.

i$)$ Text Preprocessing: $(2$ Marks$)$

Tokenization

Lowercasing

Stop Words Removal

Stemming

Lemmatization

ii$)$ Feature Extraction: $(2$ Marks$)$

Use the pre$-$processed data from previous step and implement the below vectorization methods to extract features.

Word Embedding using TF$-$IDF

iii$)$ Similarity Analysis: $(3$ Marks$)$

Use the vectorized representation from previous step and implement a method to identify and print the names of top two similar documents that exhibit significant similarity. Justify your choice of similarity metric and feature design. Visualize a subset of vector embedding in $2$D semantic space suitable for this use case. HINT: $($Use PCA for Dimensionality reduction$)$

Verify and Need solution