You are supposed to use the same dataset as used in Assignment 1 for this question.

Download the stories dataset from the given link: http://archives.textfiles.com/stories.zip 

The data set consists of 467 files and has a size of about 15MB (including SRE and remaining

files). The Farnon folder is excluded from the dataset. Ignore index.html in the stories folder.

1) Carry out the following preprocessing steps on the given dataset

i. Convert the text to lower case

ii. Perform word tokenization

iii. Remove stopwords from tokens

iv. Remove punctuation marks from tokens

v. Remove blank space tokens

b. Implement the positional index data structure

c. Provide support for the searching of phrase queries. You may assume query length to be

less than or equal to 5.