(6) PySpark and Spark MLlib (6 marks) (a) Assume the following Data Frame is defined in the PySpark shell. Write down your codes in PySpark shell to fulfil the following operations. 2 3 1 df_RD = spark.read.format("csv") .option("header","true") option("inferSchema", "true") 4 .load(SPARK_BOOK+"/data/retail-data/by-day/2010-12-01.csv") 5 df_RD.printSchema () root | -- InvoiceNo: string (nullable = true) StockCode: string (nullable = true) Description: string (nullable = true) Quantity: integer (nullable = true) InvoiceDate: timestamp (nullable = true) |-- UnitPrice: double (nullable = true) Customer ID: double (nullable = true) Country: string (nullable = true) 1 df_RD.show(2) InvoiceNo StockCode Description Quantity InvoiceDate UnitPriceCustomer ID Country 536365 85123A | WHITE HANGING HEA... 5363651 71053 | WHITE METAL LANTERN only showing top 2 rows 62010-12-01 08:26:00 6|2010-12-01 08:26:00 2.551 3.39 17850.0 United Kingdom 17850.0 United Kingdom +- Define a new DataFrame that includes on the StockCode" and "Quantity" columns of df_RD. Count the unique values in the "InvoceNo" column. (4 mark) (b) Explain the main difference between Spark Data Frame and Pandas Data Frame as data structures? (1 mark) (c) Spark MLlib is similar to Scikit-Learn in terms of APIs. But what is the most important feature of Spark MLlib? (1 mark) (6) PySpark and Spark MLlib (6 marks) (a) Assume the following Data Frame is defined in the PySpark shell. Write down your codes in PySpark shell to fulfil the following operations. 2 3 1 df_RD = spark.read.format("csv") .option("header","true") option("inferSchema", "true") 4 .load(SPARK_BOOK+"/data/retail-data/by-day/2010-12-01.csv") 5 df_RD.printSchema () root | -- InvoiceNo: string (nullable = true) StockCode: string (nullable = true) Description: string (nullable = true) Quantity: integer (nullable = true) InvoiceDate: timestamp (nullable = true) |-- UnitPrice: double (nullable = true) Customer ID: double (nullable = true) Country: string (nullable = true) 1 df_RD.show(2) InvoiceNo StockCode Description Quantity InvoiceDate UnitPriceCustomer ID Country 536365 85123A | WHITE HANGING HEA... 5363651 71053 | WHITE METAL LANTERN only showing top 2 rows 62010-12-01 08:26:00 6|2010-12-01 08:26:00 2.551 3.39 17850.0 United Kingdom 17850.0 United Kingdom +- Define a new DataFrame that includes on the StockCode" and "Quantity" columns of df_RD. Count the unique values in the "InvoceNo" column. (4 mark) (b) Explain the main difference between Spark Data Frame and Pandas Data Frame as data structures? (1 mark) (c) Spark MLlib is similar to Scikit-Learn in terms of APIs. But what is the most important feature of Spark MLlib? (1 mark)