Problem $1$: Word Count

In the next few problems, we will work with a text file that contains the complete works of William Shakespeare.

The data file using for this problem is located at: $/$FileStore$/$tables$/$shakespeare$\_$complete.txt$.$

We will begin by loading and processing the file and tokenizing the lines into individual words.

Complete the following steps in a single code cell:

$1.$ Read the contents of the file shakespeare$\_$complete.txt into an RDD named ws$\_$lines.

$2.$ Create an RDD named ws$\_$words by applying the transformations described below. This will require

several uses of map$()$ and flatMap$()$ and a single call to filter$().$ Try to chain together the

transformations together to complete all of these steps with a single statement $($that will likely span

multiple lines$).$

$$ Tokenize the strings in ws$\_$lines by splitting them on the $8$ characters in the following list:

$[\text{'}\text{'},\text{'}-\text{'},\text{'}\_\text{'},\text{'}.\text{'},\text{'},\text{'},\text{'}$:$\text{'},\text{'}|\text{'},\text{'}\backslash$t$\text{'}]$

The resulting RDD should consist of strings rather than lists of strings. This will require

multiple separate uses of flatMap$()$ and split$().$

$$ Use the Python string method strip$()$ with the punctuation string to remove common

punctuation symbols from the start and end of the tokens. Then use strip$()$ again with the

string $\text{'}0123456789\text{'}$ to remove numbers from the start and end of the tokens.

$($Code cell continued on next page.$)$

$($Code cell continued from previous page.$)$

$$ Use the Python string method replace$()$ to replaces instances of the single

quote$/$apostrophe $"\text{'}"$ with the empty string $\text{'}\text{'}.$

$$ Convert all strings to lower case using the lower$()$ string method.

$$ The steps above will create some empty strings of the form $\text{'}\text{'}$ within the RDD$.$ Filter out

these empty strings.

$3.$ Create a second RDD named dist$\_$words that contains only one copy of each word found in

ws$\_$words.

$4.$ Print the number of words in ws$\_$words and the number of distinct words using the format shown

below. Add spacing so that the numbers are left$-$aligned.

Total Number of Words: xxxx

Number of Distinct Words: xxxx

We will now use sample$()$ to get a sense as to the types of words found in ws$\_$word