DSCI $617$ HW $02$ Instructions

General Instructions

Navigate to the Homework folder inside of your user directory in the Databricks workspace. Create a notebook

named HW$\_02$ inside the Homework folder.

Any set of instructions you see in this document with an orange bar to the left will indicate a place where you

should create a markdown cell. For each new problem, create a markdown cell that indicates the title of that

problem as a level $2$ header. Any set of instructions you see with a blue bar to the left will provide instructions

for creating a single code cell.

Add a markdown cell that displays the following text as a level $1$ header: DSCI $617$ Homework $02.$ Within the

same cell, on the line below the header, add your name in bold.

Add a code cell that imports the SparkSession class and the pandas package. Use the standard alias for

pandas. Also import the punctuation string from the string library.

Add another code cell to create SparkSession and sparkContext objects named spark and sc$.$

Problem $1$: Word Count

In the next few problems, we will work with a text file that contains the complete works of William Shakespeare.

The data file using for this problem is located at: $/$FileStore$/$tables$/$shakespeare$\_$complete.txt$.$

We will begin by loading and processing the file and tokenizing the lines into individual words.

Complete the following steps in a single code cell:

$1.$ Read the contents of the file shakespeare$\_$complete.txt into an RDD named ws$\_$lines.

$2.$ Create an RDD named ws$\_$words by applying the transformations described below. This will require

several uses of map$()$ and flatMap$()$ and a single call to filter$().$ Try to chain together the

transformations together to complete all of these steps with a single statement $($that will likely span

multiple lines$).$

$$ Tokenize the strings in ws$\_$lines by splitting them on the $8$ characters in the following list:

$[\text{'}\text{'},\text{'}-\text{'},\text{'}\_\text{'},\text{'}.\text{'},\text{'},\text{'},\text{'}$:$\text{'},\text{'}|\text{'},\text{'}\backslash$t$\text{'}]$

The resulting RDD should consist of strings rather than lists of strings. This will require

multiple separate uses of flatMap$()$ and split$().$

$$ Use the Python string method strip$()$ with the punctuation string to remove common

punctuation symbols from the start and end of the tokens. Then use strip$()$ again with the

string $\text{'}0123456789\text{'}$ to remove numbers from the start and end of the tokens.

$($Code cell continued on next page.$)$

$($Code cell continued from previous page.$)$

$$ Use the Python string method replace$()$ to replaces instances of the single

quote$/$apostrophe $"\text{'}"$ with the empty string $\text{'}\text{'}.$

$$ Convert all strings to lower case using the lower$()$ string method.

$$ The steps above will create some empty strings of the form $\text{'}\text{'}$ within the RDD$.$ Filter out

these empty strings.

$3.$ Create a second RDD named dist$\_$words that contains only one copy of each word found in

ws$\_$words.

$4.$ Print the number of words in ws$\_$words and the number of distinct words using the format shown

below. Add spacing so that the numbers are left$-$aligned.

Total Number of Words: xxxx

Number of Distinct Words: xxxx

We will now use sample$()$ to get a sense as to the types of words found in ws$\_$words.

Draw a sample from ws$\_$words using the arguments withReplacement$=$False and fraction$=0\mathrm{.}0001.$

Collect and print the results.

Problem $2$: Longest Words

We will now find the longest words used by Shakespeare. We will start by looking for the single longest word.

Complete the following steps in a single code cell:

$1.$ Write a Python function with two parameters, both of which are intended to be strings. The function

should return the longer of the two strings. If the strings are the same length, then the function

should return the word that appears later when ordered lexicographically $($alphabetically$).$

$2.$ Use the function you wrote along with reduce$()$ to find the longest word in the RDD dist$\_$words.

Print the result.

We will now find the $20$ longest words used by Shakespeare.

Use sortBy$()$ with the Python len$()$ function to sort the elements of dist$\_$words according to their

length, with longer words appearing first. Print the first $20$ elements of this RDD$.$

Problem $3$: Word Frequency

We will now create a frequency distribution for the words appearing in our document in order to determine

which words were used most frequently by Shakespeare.

Complete the following steps in a single code cell:

$1.$ Create an RDD named pairs. This RDD should consist of tuples of the form $($x$,1),$ where x is a

word in ws$\_$words. The RDD pairs should contain one element for each element of ws$\_$words.

$2.$ Use reduceByKey$()$ to group the pairs together according to their first elements $($the words$),$

summing together the integers stored in the second element $($the $1$s$).$ This will produce an RDD with

one pair for each distinct word. The first element will be the word and the second element will be a

count for that word. Sort this RDD by the second tuple element $($the count$),$ in descending order.

Name the resulting RD