Using Python Spark RDD for data processing :

Question 1: Counts the number of words of each length: Your program should return the number of 1-letter words, of 2-letter words, and so on. Run your program on pg100.txt file. What is the average word size? (This program should work on any random text file of any size) Question 2: Count the frequency of each letter in a file. Run your program using the pg100.txt file. Show the five most used letters. (This program should work on any random text file of any size) Question 3: See the attached input file OrderDB.txt. Each line in this file contains {Order-ID, Customer_id, Order_date, total} where the total is the amount of money spent by the customer on that order. Write a Spark program that will output the most expensive order per customer. Question 4: Show the total amount spent by each customer considering all the orders.

** The entire text file is not included due to its size.

pg100.txt (**5.33MB):

The Project Gutenberg EBook of The Complete Works of William Shakespeare,

by

William Shakespeare

This eBook is for the use of anyone anywhere at no cost and with

almost no restrictions whatsoever. You may copy it, give it away or

re-use it under the terms of the Project Gutenberg License included

with this eBook or online at www.gutenberg.org

** This is a COPYRIGHTED Project Gutenberg eBook, Details Below **

** Please follow the copyright guidelines in this file. **

Title: The Complete Works of William Shakespeare

Author: William Shakespeare

Posting Date: September 1, 2011 [EBook #100]

Release Date: January, 1994

Language: English

*** START OF THIS PROJECT GUTENBERG EBOOK COMPLETE WORKS--WILLIAM

SHAKESPEARE ***

Produced by World Library, Inc., from their Library of the Future

This is the 100th Etext file presented by Project Gutenberg, and

is presented in cooperation with World Library, Inc., from their

Library of the Future and Shakespeare CDROMS. Project Gutenberg

often releases Etexts that are NOT placed in the Public Domain!!

Shakespeare

* This Etext has certain copyright implications you should read!*

<

SHAKESPEARE IS COPYRIGHT 1990-1993 BY WORLD LIBRARY, INC., AND IS

PROVIDED BY PROJECT GUTENBERG ETEXT OF ILLINOIS BENEDICTINE COLLEGE

WITH PERMISSION. ELECTRONIC AND MACHINE READABLE COPIES MAY BE

DISTRIBUTED SO LONG AS SUCH COPIES (1) ARE FOR YOUR OR OTHERS

PERSONAL USE ONLY, AND (2) ARE NOT DISTRIBUTED OR USED

COMMERCIALLY. PROHIBITED COMMERCIAL DISTRIBUTION INCLUDES BY ANY

SERVICE THAT CHARGES FOR DOWNLOAD TIME OR FOR MEMBERSHIP.>>

* Project Gutenberg is proud to cooperate with The World Library*

in the presentation of The Complete Works of William Shakespeare

for your reading for education and entertainment. HOWEVER, THIS

OrderDB.txt (**418KB):

o-NSbj6gDPPV,c-1bB3dD1bB,1327519190,338.0

o-bajvS8vqL0,c-2cC1bB3dD,1326614203,396.46

o-LH3up9m3GQ,c-4eE4eE4eE,1325745074,421.65

o-wgJA3a5qgX,c-2cC1bB0aA,1326166422,669.83

o-NHLHPzHoD3,c-0aA5fF4eE,1326838521,531.54

o-CZVTpibVpF,c-2cC2cC4eE,1325722860,137.33

o-qaknWo1SDE,c-5fF5fF5fF,1326892904,186.18

o-64q6HcIf9k,c-2cC0aA2cC,1326654479,836.86

o-yalb0BMcGW,c-5fF0aA1bB,1325966229,70.0

o-xBREWAgOBV,c-0aA5fF4eE,1325911979,704.55

o-ZM4JLwRbzx,c-3dD2cC2cC,1326056603,982.58

o-0BUHdV8zQw,c-3dD5fF3dD,1326550921,225.31

o-rKND47TKUq,c-5fF2cC5fF,1327454854,397.12

o-0aCyM4lVYV,c-1bB5fF1bB,1325435919,671.64

o-uxcDFZSD04,c-5fF0aA0aA,1326923556,779.77

o-evgnQl9sDW,c-1bB2cC3dD,1325831954,349.74

o-3L8hk9ZREn,c-4eE3dD5fF,1326337611,226.28

o-lAq57oYiI8,c-2cC1bB5fF,1327090460,563.63

o-1ybKZRrZyS,c-0aA1bB1bB,1327184195,403.24

o-ykPEbmVVMT,c-1bB0aA0aA,1325659974,395.82

o-W20I2dt1Dh,c-5fF2cC1bB,1326953021,454.57

o-UnzpDWOOww,c-0aA1bB3dD,1326657774,939.1

o-I1pZ8n6VTE,c-1bB0aA1bB,1327303045,893.03

o-C0rW8Ap4kX,c-3dD5fF1bB,1325540201,181.53

o-dkIFTL5ZH9,c-4eE0aA2cC,1326233065,231.3

o-xYJpNL6j9d,c-5fF4eE3dD,1327268966,680.6

o-Zr2N0eIlRl,c-3dD5fF2cC,1326694425,412.88

o-rODfLMWfxd,c-2cC0aA1bB,1327189007,761.74

o-iK1VG8OdTo,c-2cC3dD4eE,1327238578,26.83

o-DtnstlrTEf,c-3dD5fF3dD,1327471598,187.71

o-FY5Ejh5LH1,c-4eE3dD5fF,1325980006,521.14

o-f16LvQBNIw,c-0aA0aA1bB,1326504317,654.2

o-0e2aK9DY8l,c-2cC1bB3dD,1326404188,484.77

o-qhZpBtavNA,c-4eE0aA4eE,1326078817,720.54

o-FWPOlD4ZUN,c-0aA2cC1bB,1325777374,872.4

o-07gsNpO2Kk,c-4eE2cC1bB,1327206673,784.72

o-Xkp2ksM5ca,c-0aA5fF1bB,1327149588,786.68

o-K5lzZ8mmhT,c-3dD5fF0aA,1326139330,606.56

o-uWtz1DC3Kf,c-1bB2cC1bB,1326197337,987.78

o-og4HmWHeT9,c-4eE1bB1bB,1325624444,303.25

o-9zNjxRmrRU,c-1bB0aA2cC,1326034059,542.65

o-bPm6htrJZK,c-0aA1bB1bB,1325667225,104.44

o-auCC4wuxRv,c-0aA5fF2cC,1326961524,418.34

o-rhmElz4Npo,c-2cC1bB0aA,1326361671,485.37

o-KizsYTSkE3,c-3dD4eE4eE,1325896751,764.17

o-Zp02fuNB0g,c-5fF4eE2cC,1326881302,265.78

o-A38EVwcj8j,c-2cC1bB0aA,1325620160,242.3

o-7mhsKNQvLI,c-0aA3dD0aA,1326171498,560.85

o-MiGIraU90h,c-2cC1bB5fF,1325579620,255.46

o-1TDdy8iFuY,c-3dD3dD2cC,1327366393,900.12

o-Q41x1DrXcf,c-2cC0aA1bB,1327219923,540.14

o-WqIieGW2Ha,c-3dD4eE1bB,1326337940,415.5

o-4lXzgc5hok,c-2cC5fF3dD,1326725406,883.15

o-nBHdc1qPj1,c-2cC0aA2cC,1327327237,779.24

o-homQmjmVMN,c-3dD0aA2cC,1325566700,999.32

o-2sce40odb8,c-0aA3dD0aA,1326808199,703.54

o-sB5LjE0etx,c-0aA4eE2cC,1325915868,249.81

o-zz2tI4fiuU,c-4eE4eE5fF,1325652987,390.62

o-Kz82ao1eiy,c-5fF3dD4eE,1326602601,975.46

o-3JJv5W3c7f,c-4eE1bB4eE,1325617062,580.13

o-mNFmgHlEJg,c-2cC0aA5fF,1326948604,311.08

o-9IEmxqnszc,c-4eE2cC2cC,1326003604,521.21

o-r1pS9dxlF6,c-3dD5fF3dD,1327268307,9.15

o-AkrESjhzR0,c-5fF1bB5fF,1325888182,303.23

o-lCI2fmwx0X,c-0aA3dD0aA,1326050538,3.12

o-8mTEqOroKz,c-3dD1bB0aA,1327531517,46.7

o-enh5yLvCOR,c-3dD2cC2cC,1325976974,687.4

o-3KtOThdT9f,c-0aA0aA5fF,1327058819,766.21

o-5Wn6eesP2Z,c-3dD4eE2cC,1325766300,153.81

o-B7k6Hagz7e,c-5fF1bB2cC,1327485638,523.16

o-HfpqPsZnyw,c-2cC2cC2cC,1327074640,35.74

o-0Y5ZIf9fdS,c-5fF0aA3dD,1327494603,6.2

o-EcDxCE046f,c-5fF2cC3dD,1325886139,494.7

o-litMqY5bvJ,c-2cC5fF3dD,1325581795,492.15

o-Kf6kd76i0s,c-3dD3dD3dD,1325569271,593.23

o-dXETpp9CsD,c-1bB4eE2cC,1327444901,877.16

o-oACNaEZ1uN,c-4eE2cC0aA,1326253302,132.35

o-vt6aJe4q9p,c-2cC1bB2cC,1326156205,606.63

o-sEWKqzhynL,c-5fF3dD4eE,1325769596,658.63

o-Wa7jV9Lek3,c-2cC2cC2cC,1327199884,960.8

o-nz0Go4Y36f,c-2cC0aA2cC,1326420667,568.88

o-GW6oAmplKC,c-4eE5fF0aA,1326896661,887.15

o-Z0tOh1zSaH,c-1bB0aA5fF,1326721913,601.37

o-r294W02Hnx,c-4eE4eE0aA,1325498673,655.47

o-H60EQwCwz3,c-4eE2cC2cC,1326897386,905.36

o-IJNOmNOFdK,c-3dD0aA5fF,1326652962,909.2

o-2VshLoS6bp,c-4eE3dD2cC,1326675243,76.3

o-Z7M0LvV0bY,c-0aA4eE1bB,1326931070,438.17

o-tlhfqggQJt,c-4eE3dD0aA,1327276217,98.02

o-p4ZDnCtVtm,c-4eE5fF4eE,1326049549,192.61

o-tf5k9wUgRF,c-2cC5fF3dD,1326969171,375.4

o-AIdDw3O52v,c-2cC2cC2cC,1326232340,770.68

o-ZMrjMuvS0o,c-1bB4eE3dD,1326652435,667.57

o-09Ox6l8PKB,c-1bB3dD3dD,1326584605,256.21

o-80wDNmqSmD,c-5fF4eE2cC,1327186700,510.27

o-7N2q4bbrW4,c-2cC4eE1bB,1325528731,970.53

o-NArlLdSgqe,c-4eE4eE0aA,1326622442,581.63

o-idDJTRJNYE,c-2cC2cC5fF,1326600360,833.4

o-11QsaODe2o,c-4eE5fF1bB,1327133834,205.04

o-QcN10UWIyo,c-5fF2cC5fF,1325706776,255.8

o-MUA4yrdsAS,c-1bB0aA2cC,1325722069,372.71

o-YeAR5irW7Q,c-4eE3dD5fF,1326277098,15.52

o-anAAtlvdnY,c-2cC2cC0aA,1326701544,97.27

o-2QRI4Dbalt,c-2cC1bB3dD,1326549537,874.31

o-BcWljQAHAT,c-2cC4eE4eE,1325928326,575.68

o-g0sLFMsVU9,c-3dD5fF0aA,1326766078,152.83

o-qyhD2dWZmN,c-5fF4eE3dD,1327519652,540.31