Problem $4$: Calculating SSE

In this problem, we will read in and process a data file containing two decimal values in each line. The first value

is intended to represent an observed value of some random variable, while the second value is intended to

represent a prediction generated by some model. We will use these values to calculate the sum of squared

errors score for the predictions.

The path for the data file we will be using in this problem is $/$FileStore$/$tables$/$pairs$\_$data.txt$.$ We will

start by reading the data file and counting the number of records.

Complete the following steps in a single code cell:

Read the contents of the data file into and RDD named pairs$\_$raw.

Display the number of elements contained in the pairs$\_$raw RDD

We will now display the first few elements of this RDD$.$

Use a for loop and the take$()$ method to display the first $5$ elements of pairs$\_$raw. Note that these

elements are stored as strings.

We will now process each of the elements of the RDD by tokenizing each string and coercing the individual

values to floats.

Complete the following steps in a single code cell:

Write a function named process$\_$line$().$ The function should accept a single parameter named

row. This parameter is intended to take on string values of the type stored in pairs$\_$raw. The

function should split the string at the space character, coerce each of the two tokens into float

values, and return a tuple containing these two float values.

Use the map$()$ transformation to apply process$\_$line$()$ to pairs$\_$raw storing the resulting RDD

in pairs.

Use a for loop and the take$()$ method to display the first $5$ elements of pairs.

We will now calculate the sum of squared errors score for the values stored in the RDD we have created.

Need help in python databricks: Complete the following steps in a single code cell:

Use map$()$ with a lambda function to calculate the squared difference of each pair of values stored

in pairs. Then call the sum$()$ method of the resulting RDD$,$ storing the result in a variable named

SSE.

Print the value of SSE.