The CiteSeer UMD collection is a standard text document collection, consisting of abstracts of research articles from Computer Science, which are sampled from the CiteSeer digital library. The dataset is available for download from blackboard. Tasks: 1. Write a program that preprocesses the collection. This preprocessing stage should specifically include a function that tokenizes the text. In doing so, tokenize on whitespace and remove punctuation. For this task, please use your own implementation of a tokenizer. 2. Determine the frequency of occurrence for all the words in the collection. Answer the following questions: 1. What is the total number of words in the collection? 2. What is the vocabulary size? (i.e., number of unique terms). 3. What are the top 20 words in the ranking? (i.e., the words with the highest frequencies). 4. From these top 20 words, which ones are stop-words? 5. What is the minimum number of unique words accounting for 15% of the total number of words in the collection? Example: if the total number of words in the collection is 100, and we have the fol- lowing word-frequency pairs: Word the of data mining the answer to this question will be (1 word accounts for 15% of the total 100 words). 3. Integrate the Porter stemmer and a stopword eliminator into your code. Answer again questions a.-e. from the previous point. (See below a link to a Java Porter stemmer implementation and to a stopwords list). Note: It is highly recommended that your code is as modularized as possible; many of the functions that you implement during this assignment will be needed in future assignments or in the term project. Submission instructions: 1. write a README file including: . a detailed note about the functionality of each of the above programs, complete instructions on how to run them answers to the questions above 2. make sure you include your name in each program and in the README file. 3. make sure all your programs run correctly on the CSE machines. . Verizon 10:30 AM C@ 75% HW 1.pdf The Citur UMD collection is a standard est document collection consisting of ach articles from Computer Science, which are sampled from the dat is available for download from Blackboard acts of heary. The d Write a program that was the c specifically include a function that hi these t l w ee 2 Der the frequency o ccurrence for all the words in the collection the 2. What is the vocabulary , sumber of ti m e What are the top 20 words in the ranking the words with the highest frequencies From the top 20 words, which ones are top-words? 5. What is the immer geworden for 15th Emple the ot h er of words in the collection is 100, and we have the lewing word frequency pain the answer to this question will be (1 word accounts for 1 of the total 100 words) 3. Integrate the Porter stemmer and a stopword eliminator into your code Answer again questions are from the previous points below a link to a lava Porter emmer implementation and to stopwords list) p No highly ended that your com that you implement during this will be seeded in o o und the t wa README detailed the functie of each of the program 2.mkewe you include your cachogram and in the README . make sure all your program correctly on the machines The CiteSeer UMD collection is a standard text document collection, consisting of abstracts of research articles from Computer Science, which are sampled from the CiteSeer digital library. The dataset is available for download from blackboard. Tasks: 1. Write a program that preprocesses the collection. This preprocessing stage should specifically include a function that tokenizes the text. In doing so, tokenize on whitespace and remove punctuation. For this task, please use your own implementation of a tokenizer. 2. Determine the frequency of occurrence for all the words in the collection. Answer the following questions: 1. What is the total number of words in the collection? 2. What is the vocabulary size? (i.e., number of unique terms). 3. What are the top 20 words in the ranking? (i.e., the words with the highest frequencies). 4. From these top 20 words, which ones are stop-words? 5. What is the minimum number of unique words accounting for 15% of the total number of words in the collection? Example: if the total number of words in the collection is 100, and we have the fol- lowing word-frequency pairs: Word the of data mining the answer to this question will be (1 word accounts for 15% of the total 100 words). 3. Integrate the Porter stemmer and a stopword eliminator into your code. Answer again questions a.-e. from the previous point. (See below a link to a Java Porter stemmer implementation and to a stopwords list). Note: It is highly recommended that your code is as modularized as possible; many of the functions that you implement during this assignment will be needed in future assignments or in the term project. Submission instructions: 1. write a README file including: . a detailed note about the functionality of each of the above programs, complete instructions on how to run them answers to the questions above 2. make sure you include your name in each program and in the README file. 3. make sure all your programs run correctly on the CSE machines. . Verizon 10:30 AM C@ 75% HW 1.pdf The Citur UMD collection is a standard est document collection consisting of ach articles from Computer Science, which are sampled from the dat is available for download from Blackboard acts of heary. The d Write a program that was the c specifically include a function that hi these t l w ee 2 Der the frequency o ccurrence for all the words in the collection the 2. What is the vocabulary , sumber of ti m e What are the top 20 words in the ranking the words with the highest frequencies From the top 20 words, which ones are top-words? 5. What is the immer geworden for 15th Emple the ot h er of words in the collection is 100, and we have the lewing word frequency pain the answer to this question will be (1 word accounts for 1 of the total 100 words) 3. Integrate the Porter stemmer and a stopword eliminator into your code Answer again questions are from the previous points below a link to a lava Porter emmer implementation and to stopwords list) p No highly ended that your com that you implement during this will be seeded in o o und the t wa README detailed the functie of each of the program 2.mkewe you include your cachogram and in the README . make sure all your program correctly on the machines
