Answered step by step
Verified Expert Solution
Link Copied!

Question

1 Approved Answer

Please help me with my code, is it because of the TimestamRTVRe of data? The QUERY WHERE and ORDER BY does not work. The WHERE

Please help me with my code, is it because of the TimestamRTVRe of data? The QUERY
WHERE and ORDER BY does not work. The WHERE first part without the ORDER BY works.
Consider a warc.csv file related data. An indicative line is:
219.155.1.48, http:o000, Apaohe, 'chtml>'
Columns in order: first the WarS date, the warS record id, the WerS type (e.g. metadata,
response, etc), the content length, the public IP address, the target URL, the server running
the site (eg apache, DginX, etc), and finally the overall content of the page with the entire
HTML DOM. For the time range between 2017-03-2222:00 and 2017-03-2223:00, find the 5
most used servers. Results to be given in descending order of servers.
from datetime import datetime
from Ryspark, sol import SoarkSession
from pysparksaltypes import StructType, StructField, StringTvo, IntegerType,
FlgatTxpe TimestamoTyRE.
# Initialize Spark Session
spark = SparkSession.byilderaRRName("Warcnalysis")-getQrCreate()
# Define the schema
schema = StructTXRel [
StructField["date", TimestamaTVRed), True),
StructField["recordid", StringTyRed, True),
StructField("type", StringType(), True),
StructField ["contentalength", IntegerType(), True),
StructField("RublicuiR", StringTvRe(), True),
StructField("target,url", StriggTvag0), True),
StructField ("server", StriogTvRe(), True),
StructField ("btmldom", StriogTvae(), True)
# Load the data into RataFrame
df= sparkread format('csv')}
options(header='false')\
schema(schema)\
load("warc.csv")
# Register the Bata Frame as a temporary table
df.reateOrReglaceTempView("warc")
##Filter the data using Spark SQL
id quex = "SELECT yarcseryer
FROM wars \
WHERE Yarc.date =?12017-03-22T22:00:00Z'' AND warcodate '2017-03-
22T22:00:00Z
ORDER BY warcodate ASC ?n
filteced,df = spack.sgl(id,guery)
# Show the result
filteced df.shou(5)
# Stop Spark Session
sparkst8R()

Step by Step Solution

There are 3 Steps involved in it

Step: 1

blur-text-image

Get Instant Access to Expert-Tailored Solutions

See step-by-step solutions with expert insights and AI powered tools for academic success

Step: 2

blur-text-image

Step: 3

blur-text-image

Ace Your Homework with AI

Get the answers you need in no time with our AI-driven, step-by-step assistance

Get Started

Recommended Textbook for

Data And Information Quality Dimensions, Principles And Techniques

Authors: Carlo Batini, Monica Scannapieco

1st Edition

3319241060, 9783319241067

More Books

Students also viewed these Databases questions