Answered step by step
Verified Expert Solution
Link Copied!

Question

1 Approved Answer

How do you specify the level of compression you want writing to HDFS using avro snappy compression in the language spark scala? I want to

How do you specify the level of compression you want writing to HDFS using avro snappy compression in the language spark scala?

I want to compress it to 1.

this is what i have so far:

sqlContext.setConf("spark.sql.avro.compression.codec","snappy")

sqlContext.setConf("spark.sql.avro.compression.snappy","1")

result.write.avro("/user/cloudera/products/result")

Would this work or are you not allowed to pick what you compress it down to?

Step by Step Solution

There are 3 Steps involved in it

Step: 1

blur-text-image

Get Instant Access to Expert-Tailored Solutions

See step-by-step solutions with expert insights and AI powered tools for academic success

Step: 2

blur-text-image

Step: 3

blur-text-image

Ace Your Homework with AI

Get the answers you need in no time with our AI-driven, step-by-step assistance

Get Started

Recommended Textbook for

Intelligent Databases Technologies And Applications

Authors: Zongmin Ma

1st Edition

1599041219, 978-1599041216

More Books

Students also viewed these Databases questions