roblem $3$: Diamonds Data $($Part $1)$

For the remainder of this assignment, we will again be working with the Diamonds dataset. Recall that the data

file for this dataset is located at the path: $/$FileStore$/$tables$/$diamonds$.$txt

Recall also that each row of this data file contains the following information for a single diamond:

$$ carat Weight of the diamond.

$$ cut Quality of the cut. Levels: Fair, Good, Very Good, Premium, Ideal

$$ color Diamond color. Levels: J$,$ I, H$,$ G$,$ F$,$ E$,$ D

$$ clarity A measure of diamond clarity. Levels: I$1,$ SI$2,$ SI$1,$ VS$2,$ VS$1,$ VVS$2,$ VVS$1,$ IF

$$ depth Total depth percentage

$$ table Width of top of diamond relative to widest point

$$ price Price in US dollars

$$ x Length in mm

$$ y Width in mm

$$ z Depth in mm

We will begin by loading the dataset into a DataFrame. For the purposes of creating a schema, note that the

price column contains integer values. The cut, color, and clarity columns each contain string values. All

other columns contain floating point $($or double$)$ values.

Create a code cell to complete the following tasks:

$1.$ Create a custom schema for this dataset. I recommend using a DDL string.

$2.$ Read the file into a DataFrame named diamonds using spark.read$().$ Note that the file is tabdelimited and has a header.

$3.$ Use printSchema$()$ to display the DataFrame's schema.

Next, we will calculate the number of records in in the dataset.

Create a new code cell and use it to display the number of rows in the diamonds DataFrame.

To get a sense as to what the data looks like, we will display the first $10$ rows of the DataFrame.

Use the show$()$ method to display the first $10$ rows of the diamonds DataFrame.

Lastly, we will explore the relationship between carat size and diamond price by drawing a sample from the

DataFrame and then generating a plot of price versus carat size using the observations in the sample.

Create a code cell to complete the following tasks:

$1.$ Use the sample$()$ method to draw a sample from diamonds. Use fraction$=0\mathrm{.}25$ and seed$=1.$

Convert the sample to a Pandas DataFrame and store the result in sample$\_$pdf$.$

$2.$ Use the data in the sample to create a scatter plot of price versus carat. When creating the scatter

plot, set alpha$=0\mathrm{.}5$ and select a named color for the points. Label the x$-$axis "Carat" and label the

y$-$axis "Price". Use plt$.$show$()$ to display the plot.

Note that we could have easily plotted the entire dataset in the cell above, rather than limiting ourselves to a

sample. However, when working with very large dataset in a distributed setting, this is not always feasible. The

process above can be used in these situations.