Answered step by step
Verified Expert Solution
Link Copied!

Question

1 Approved Answer

Python code - Recieving error messages :rg.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 82.0 failed 1 times, most recent failure: Lost

Python code - Recieving error messages

:rg.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 82.0 failed 1 times, most recent failure: Lost task 0.0 in stage 82.0 (TID 126) (ip-10-172-228-171.us-west-2.compute.internal executor driver): org.apache.spark.api.python.PythonException: 'ValueError: could not convert string to float: '2.080750 1.099070 0.927763 1.029080 -0.130763 1.265460 -0.436286 0.728881 2.107980 1.359870 0.265471 -0.324436 -1.200930 -1.531920 0.298026 0.088341 0.644631 -0.650944 -0.891003 -0.185717 1.395990 -0.832146 2.115500 1.347400 0.319671 0.206106 -0.377173 -1.607610 0.856661 -0.208192 -0.227354 0.455312 0.700828 1.460820 -0.127914 1.686890 -1.400370 1.306270 0.640420 0.640420 0.640420 0.640420 0.584761 0.777720 0.946010 1.448660 -0.016243 -0.516636 0.150824 1.572940 1.572940 1.572940 1.572940 -0.631182 0.252795 -0.355982 -1.897550 -0.433645 1.994650 -0.018567 0.061276 -0.604689 0.905910 -0.782365 0.964356 0.016980 0.116946 -0.733182 -0.468315 -0.361302 1.591210 -0.734161 0.652221 0.406108 1.462260 -0.487164 -0.457068 -0.158002 -0.356138 -0.680514 0.747648 -1.662690 -0.925094 -2.291300 -0.165720

The code works fine on a smaller file that has comma to separate the data. Any suggesting?

CODE

import pandas as pd

from pyspark.mllib.stat import Statistics ##Dataframe

df = spark.read.format("csv").option("header", "False",).load("dbfs:/FileStore/tables/colon_cancer_no_labels.txt")

data2 = df.rdd.map(list)

## to display full correlation dataframe

print(Statistics.corr(data2, method = "pearson"))

Step by Step Solution

There are 3 Steps involved in it

Step: 1

blur-text-image

Get Instant Access to Expert-Tailored Solutions

See step-by-step solutions with expert insights and AI powered tools for academic success

Step: 2

blur-text-image

Step: 3

blur-text-image

Ace Your Homework with AI

Get the answers you need in no time with our AI-driven, step-by-step assistance

Get Started