Question
We want to count the number of tweets published in each state. The following table shows the desired results: Georgia - 3 Florida - 2
We want to count the number of tweets published in each state. The following table shows the desired results:
Georgia - 3 Florida - 2 Alabama - 2 California - 4
Design and implement a PySpark program to solve the problem. You are required to use Spark Dataframe to implement this function.
Below are tweets.json, and cityStateMap.json to read from.
tweets.json:
[{
"user": "Bob",
"geo": "Atlanta",
"tweet": "It is a sunny day!"
},
{
"user": "Susan",
"geo": "Athens",
"tweet": "We have a football game today :)"
},
{
"user": "David",
"geo": "Atlanta",
"tweet": "Today is cold."
},
{
"user": "Lisa",
"geo": "Auburn",
"tweet": "I love Auburn University"
},
{
"user": "Ben",
"geo": "Birmingham",
"tweet": "I will go to Atlanta today!"
},
{
"user": "Paul",
"geo": "San Francisco",
"tweet": "We watch a movie today!"
},
{
"user": "Smith",
"geo": "San Diego",
"tweet": "It is hot today. Summer is here."
},
{
"user": "Ethan",
"geo": "Log Angeles",
"tweet": "Oscar ceremony is wonderful!"
},
{
"user": "Emma",
"geo": "Log Angeles",
"tweet": "I love Oscar ceremony!"
},
{
"user": "Rolando",
"geo": "Orlando",
"tweet": "I will go to the beach!"
},
{
"user": "Mia",
"geo": "Miami",
"tweet": "Sunny Day!"
}]
cityStateMap.json:
[{
"city": "Atlanta",
"state": "Georgia"
},
{
"city": "Athens",
"state": "Georgia"
},
{
"city": "Miami",
"state": "Florida"
},
{
"city": "Orlando",
"state": "Florida"
},
{
"city": "Birmingham",
"state": "Alabama"
},
{
"city": "Auburn",
"state": "Alabama"
},
{
"city": "Log Angeles",
"state": "California"
},
{
"city": "San Francisco",
"state": "California"
},
{
"city": "San Diego",
"state": "California"
}]
A screenshot of sample output for reference:
Step by Step Solution
There are 3 Steps involved in it
Step: 1
Get Instant Access to Expert-Tailored Solutions
See step-by-step solutions with expert insights and AI powered tools for academic success
Step: 2
Step: 3
Ace Your Homework with AI
Get the answers you need in no time with our AI-driven, step-by-step assistance
Get Started
