We want to count the number of tweets published in each state. The following table shows the desired results:

Georgia - 3 Florida - 2 Alabama - 2 California - 4

Design and implement a PySpark program to solve the problem. You are required to use Spark Dataframe to implement this function.

Below are tweets.json, and cityStateMap.json to read from.

tweets.json:

[{

"user": "Bob",

"geo": "Atlanta",

"tweet": "It is a sunny day!"

},

{

"user": "Susan",

"geo": "Athens",

"tweet": "We have a football game today :)"

},

{

"user": "David",

"geo": "Atlanta",

"tweet": "Today is cold."

},

{

"user": "Lisa",

"geo": "Auburn",

"tweet": "I love Auburn University"

},

{

"user": "Ben",

"geo": "Birmingham",

"tweet": "I will go to Atlanta today!"

},

{

"user": "Paul",

"geo": "San Francisco",

"tweet": "We watch a movie today!"

},

{

"user": "Smith",

"geo": "San Diego",

"tweet": "It is hot today. Summer is here."

},

{

"user": "Ethan",

"geo": "Log Angeles",

"tweet": "Oscar ceremony is wonderful!"

},

{

"user": "Emma",

"geo": "Log Angeles",

"tweet": "I love Oscar ceremony!"

},

{

"user": "Rolando",

"geo": "Orlando",

"tweet": "I will go to the beach!"

},

{

"user": "Mia",

"geo": "Miami",

"tweet": "Sunny Day!"

}]

cityStateMap.json:

[{

"city": "Atlanta",

"state": "Georgia"

},

{

"city": "Athens",

"state": "Georgia"

},

{

"city": "Miami",

"state": "Florida"

},

{

"city": "Orlando",

"state": "Florida"

},

{

"city": "Birmingham",

"state": "Alabama"

},

{

"city": "Auburn",

"state": "Alabama"

},

{

"city": "Log Angeles",

"state": "California"

},

{

"city": "San Francisco",

"state": "California"

},

{

"city": "San Diego",

"state": "California"

}]

A screenshot of sample output for reference:

geo tweetl userl Atlanta| It is a sunny day!| Bobl Athens |We have a footbal...| Susan | Today is cold. David| Atlanta| Auburn| I love Auburn Uni...| Lisa I Birmingham| I will go to Atla...| Ben San Francisco |We watch a movie ...| PaulI I San Diego | It is hot today. ...I Smith| | LogAngeles lOscar ceremony is...I Ethan l | Log Angeles |I love Oscar cere...|Emma | Orlando |I will go to the |Rolando Sunny Day!Mia | Miamil cityl statel Atlanta| Georgia | Athens| Georgia| Miami Florida | Orlandol Floridal I Birmingham| Alabama| Auburn| Alabama| I Log Angeles |Californial San Francisco California | San Diego|California| geo tweet user state Atlantal It is a sunny day! Bob Georgial Athens |We have a footbal... Susan Georgial Today is cold. David Georgial Auburn|I love Auburn Uni...I iAlabama | | Birmingham 11 will go to Atla I Ben! Alabama 1 San Francisco |We watch a movie ...| Paul|California| I San Diego | It is hot today. ...I Smith|Californial | Log Angeles |Oscar ceremony is... | Ethan | California| I Log Angeles |I love Oscar cere...| E l Californial Orlando |I will go to the IRolandol Florida| Sunny Day Mial Florida| Atlanta| Miamil state countl Georgia 3 Alabama 2| I Floridal California 41 geo tweetl userl Atlanta| It is a sunny day!| Bobl Athens |We have a footbal...| Susan | Today is cold. David| Atlanta| Auburn| I love Auburn Uni...| Lisa I Birmingham| I will go to Atla...| Ben San Francisco |We watch a movie ...| PaulI I San Diego | It is hot today. ...I Smith| | LogAngeles lOscar ceremony is...I Ethan l | Log Angeles |I love Oscar cere...|Emma | Orlando |I will go to the |Rolando Sunny Day!Mia | Miamil cityl statel Atlanta| Georgia | Athens| Georgia| Miami Florida | Orlandol Floridal I Birmingham| Alabama| Auburn| Alabama| I Log Angeles |Californial San Francisco California | San Diego|California| geo tweet user state Atlantal It is a sunny day! Bob Georgial Athens |We have a footbal... Susan Georgial Today is cold. David Georgial Auburn|I love Auburn Uni...I iAlabama | | Birmingham 11 will go to Atla I Ben! Alabama 1 San Francisco |We watch a movie ...| Paul|California| I San Diego | It is hot today. ...I Smith|Californial | Log Angeles |Oscar ceremony is... | Ethan | California| I Log Angeles |I love Oscar cere...| E l Californial Orlando |I will go to the IRolandol Florida| Sunny Day Mial Florida| Atlanta| Miamil state countl Georgia 3 Alabama 2| I Floridal California 41