Answered step by step
Verified Expert Solution
Question
1 Approved Answer
Consider a warc.csv file related data. An indicative line is: 2 0 1 7 - 0 3 - 2 2 T 2 2 : 1
Consider a warc.csv file related data. An indicative line is:
:: record response,
adrress, URL, Server, DOM
Columns in order: first the warc date, the warc record id the warc type eg metadata,
response, etc the content length, the public IP address, the target URL, the server running
the site eg apache, nginx, etc and finally the overall content of the page with the entire
HTML DOM.
Using RDDs write a Pyhton code to answer the following.
Task :
For the target URL "coco", find the content length of the metadata as well as the size of the
HTML DOM number of characters
Tips:
For this query you should filter by url and keep warc type "request.
Remember to restart the Spark cluster before each measurement, to avoid hot caches, or
you can clear the cache with the command spark.catalog.clearCache
Task :
Perform Task using DataFramesSplark SQL and parquet file
Step by Step Solution
There are 3 Steps involved in it
Step: 1
Get Instant Access to Expert-Tailored Solutions
See step-by-step solutions with expert insights and AI powered tools for academic success
Step: 2
Step: 3
Ace Your Homework with AI
Get the answers you need in no time with our AI-driven, step-by-step assistance
Get Started