Answered step by step
Verified Expert Solution
Question
1 Approved Answer
Problem: Complete the python program file assignment2.py (included in the assignment2 folder) using spark to do the following: 1. Read the data (all the files
Problem: Complete the python program file assignment2.py (included in the assignment2 folder) using spark to do the following: 1. Read the data (all the files in the 'data' directory) using the function textFile 2. Take only the "text" part of each article and count the frequency of all the words (convert the text into lowercase) 3. Remove (Filter) any word whose frequency is less than 10 4. Report the following: 1. Total size of the output data (after the filtering) (1 point] 2. Frequency of the following words - congress, london, washington, football (1 point] 3. The word with maximum frequency for each month (hint: to read only a month's articles you can use '*. E.g. for February 2012-02*' represents all files starting with 2012-02, i.e. files belonging to Feb) (3 points) 4. List of words appeared on 2012-09-01' but not on 2012-08-01'[3 points) 5. Frequency of the word 'monsoon' for all months [2 points] Problem: Complete the python program file assignment2.py (included in the assignment2 folder) using spark to do the following: 1. Read the data (all the files in the 'data' directory) using the function textFile 2. Take only the "text" part of each article and count the frequency of all the words (convert the text into lowercase) 3. Remove (Filter) any word whose frequency is less than 10 4. Report the following: 1. Total size of the output data (after the filtering) (1 point] 2. Frequency of the following words - congress, london, washington, football (1 point] 3. The word with maximum frequency for each month (hint: to read only a month's articles you can use '*. E.g. for February 2012-02*' represents all files starting with 2012-02, i.e. files belonging to Feb) (3 points) 4. List of words appeared on 2012-09-01' but not on 2012-08-01'[3 points) 5. Frequency of the word 'monsoon' for all months [2 points]
Step by Step Solution
There are 3 Steps involved in it
Step: 1
Get Instant Access to Expert-Tailored Solutions
See step-by-step solutions with expert insights and AI powered tools for academic success
Step: 2
Step: 3
Ace Your Homework with AI
Get the answers you need in no time with our AI-driven, step-by-step assistance
Get Started