Answered step by step
Verified Expert Solution
Question
1 Approved Answer
python data bricks Problem 3 : Diamonds Data ( Part 1 ) For the remainder of this assignment, we will again be working with the
python data bricks Problem : Diamonds Data Part
For the remainder of this assignment, we will again be working with the Diamonds dataset. Recall that the data
file for this dataset is located at the path: FileStoretablesdiamondstxt
Recall also that each row of this data file contains the following information for a single diamond:
carat Weight of the diamond.
cut Quality of the cut. Levels: Fair, Good, Very Good, Premium, Ideal
color Diamond color. Levels: J I, H G F E D
clarity A measure of diamond clarity. Levels: I SI SI VS VS VVS VVS IF
depth Total depth percentage
table Width of top of diamond relative to widest point
price Price in US dollars
x Length in mm
y Width in mm
z Depth in mm
We will begin by loading the dataset into a DataFrame. For the purposes of creating a schema, note that the
price column contains integer values. The cut, color, and clarity columns each contain string values. All
other columns contain floating point or double values.
Create a code cell to complete the following tasks:
Create a custom schema for this dataset. I recommend using a DDL string.
Read the file into a DataFrame named diamonds using spark.read Note that the file is tabdelimited and has a header.
Use printSchema to display the DataFrame's schema.
Next, we will calculate the number of records in in the dataset.
Create a new code cell and use it to display the number of rows in the diamonds DataFrame.
To get a sense as to what the data looks like, we will display the first rows of the DataFrame.
Use the show method to display the first rows of the diamonds DataFrame.
Lastly, we will explore the relationship between carat size and diamond price by drawing a sample from the
DataFrame and then generating a plot of price versus carat size using the observations in the sample.
Create a code cell to complete the following tasks:
Use the sample method to draw a sample from diamonds. Use fraction and seed
Convert the sample to a Pandas DataFrame and store the result in samplepdf
Use the data in the sample to create a scatter plot of price versus carat. When creating the scatter
plot, set alpha and select a named color for the points. Label the xaxis "Carat" and label the
yaxis "Price". Use pltshow to display the plot.
Step by Step Solution
There are 3 Steps involved in it
Step: 1
Get Instant Access to Expert-Tailored Solutions
See step-by-step solutions with expert insights and AI powered tools for academic success
Step: 2
Step: 3
Ace Your Homework with AI
Get the answers you need in no time with our AI-driven, step-by-step assistance
Get Started