Machine learning with spark

Lab Support
#1

I AM RUNNING THE PROGRAM AND IT IS NOT RUNNING ,

EARLIER IT WAS SAYING THE MEMORY OUT OF SPACE , THEN I DELETED ALL MY PRVIOUS WORK AND NOW ALSO IT IS SHOWING THE ERROR AS MENTIONED BELOW

Collaborative%20Filtering%20or%20Recommender%20using%20MLlib%20_%20Automated%20hands-on_%20CloudxLab%20-%20Google%20Chrome%2028-03-2020%2017_05_58
Collaborative%20Filtering%20or%20Recommender%20using%20MLlib%20_%20Automated%20hands-on_%20CloudxLab%20-%20Google%20Chrome%2028-03-2020%2017_05_58937×588 56.5 KB

#2

Looks like there is an error somewhere in code. I am basing my judgement on “NoSuchElementException”.

Could you share the code as well as full error message?

#4

the error is comming when i am executing after the saving of mymodel

//Note: it works only in spark2
//To launch it on local mode, run the following command on cloudxlab console: spark-shell
//To launch it in yarn mode, run the following command on cloudxlab console: spark-shell --master yarn

import org.apache.spark.ml.recommendation.ALS

case class Rating(userId: Int, movieId: Int, rating: Float, timestamp: Long)
def parseRating(str: String): Rating = {
val fields = str.split("::")
assert(fields.size == 4)
Rating(fields(0).toInt, fields(1).toInt, fields(2).toFloat, fields(3).toLong)
}

//Test it
parseRating(“1::1193::5::978300760”)

var raw = sc.textFile("/data/ml-1m/ratings.dat")
//check one record. it should be res4: Array[String] = Array(1::1193::5::978300760)
//If this fails the location of file is wrong.
raw.take(1)

val ratings = raw.map(parseRating).toDF()
//check if everything is ok
ratings.show(5)

val Array(training, test) = ratings.randomSplit(Array(0.8, 0.2))

// Build the recommendation model using ALS on the training data
//Alternating Least Squares (ALS) matrix factorization.
val als = new ALS().setMaxIter(5).setRegParam(0.01).setUserCol(“userId”).setItemCol(“movieId”).setRatingCol(“rating”)

val model = als.fit(training)
model.save(“mymodel”)

//Prepare the recommendations
val predictions = model.transform(test)
predictions.map(r => r(2).asInstanceOf[Float] - r(4).asInstanceOf[Float])
.map(x => x*x)
.filter(! .isNaN)
.reduce( + _)

predictions.take(10)

predictions.write.format(“com.databricks.spark.csv”).save(“ml-predictions.csv

error message it is saying out of memmory right now also after i deleted all my files

scala> predictions.take(10)
[Stage 82:> (0 + 2) /[Stage 82:=============================> (1 + 1) /[Stage 84:> (0 + 16) / 200]20/03/28 11:34:04 ERROR Executor: Exception in task 1.0 in stage 84.0 (TID 202)
java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedOutputStream.(BufferedOutputStream.java:76)
at org.apache.spark.storage.DiskBlockObjectWriter$ManualCloseBufferedOutputStream$1.(DiskBlockObjectWriter.scala:106)
at org.apache.spark.storage.DiskBlockObjectWriter.initialize(DiskBlockObjectWriter.scala:107)
at org.apache.spark.storage.DiskBlockObjectWriter.open(DiskBlockObjectWriter.scala:115)
at org.apache.spark.storage.DiskBlockObjectWriter.write(DiskBlockObjectWriter.scala:235)
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:150)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
20/03/28 11:34:04 ERROR Executor: Exception in task 10.0 in stage 84.0 (TID 211)
java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedOutputStream.(BufferedOutputStream.java:76)
at org.apache.spark.storage.DiskBlockObjectWriter$ManualCloseBufferedOutputStream$1.(DiskBlockObjectWriter.scala:106)
at org.apache.spark.storage.DiskBlockObjectWriter.initialize(DiskBlockObjectWriter.scala:107)
at org.apache.spark.storage.DiskBlockObjectWriter.open(DiskBlockObjectWriter.scala:115)
at org.apache.spark.storage.DiskBlockObjectWriter.write(DiskBlockObjectWriter.scala:235)
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:150)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
20/03/28 11:34:04 ERROR Executor: Exception in task 13.0 in stage 84.0 (TID 214)
java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedOutputStream.(BufferedOutputStream.java:76)
at org.apache.spark.storage.DiskBlockObjectWriter$ManualCloseBufferedOutputStream$1.(DiskBlockObjectWriter.scala:106)
at org.apache.spark.storage.DiskBlockObjectWriter.initialize(DiskBlockObjectWriter.scala:107)
at org.apache.spark.storage.DiskBlockObjectWriter.open(DiskBlockObjectWriter.scala:115)
at org.apache.spark.storage.DiskBlockObjectWriter.write(DiskBlockObjectWriter.scala:235)
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:150)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
20/03/28 11:34:04 ERROR Executor: Exception in task 11.0 in stage 84.0 (TID 212)
java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedOutputStream.(BufferedOutputStream.java:76)
at org.apache.spark.storage.DiskBlockObjectWriter$ManualCloseBufferedOutputStream$1.(DiskBlockObjectWriter.scala:106)
at org.apache.spark.storage.DiskBlockObjectWriter.initialize(DiskBlockObjectWriter.scala:107)
at org.apache.spark.storage.DiskBlockObjectWriter.open(DiskBlockObjectWriter.scala:115)
at org.apache.spark.storage.DiskBlockObjectWriter.write(DiskBlockObjectWriter.scala:235)
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:150)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
20/03/28 11:34:04 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[Executor task launch worker for task 211,5,main]
java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedOutputStream.(BufferedOutputStream.java:76)
at org.apache.spark.storage.DiskBlockObjectWriter$ManualCloseBufferedOutputStream$1.(DiskBlockObjectWriter.scala:106)
at org.apache.spark.storage.DiskBlockObjectWriter.initialize(DiskBlockObjectWriter.scala:107)
at org.apache.spark.storage.DiskBlockObjectWriter.open(DiskBlockObjectWriter.scala:115)
at org.apache.spark.storage.DiskBlockObjectWriter.write(DiskBlockObjectWriter.scala:235)
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:150)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
20/03/28 11:34:04 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[Executor task launch worker for task 202,5,main]
java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedOutputStream.(BufferedOutputStream.java:76)
at org.apache.spark.storage.DiskBlockObjectWriter$ManualCloseBufferedOutputStream$1.(DiskBlockObjectWriter.scala:106)
at org.apache.spark.storage.DiskBlockObjectWriter.initialize(DiskBlockObjectWriter.scala:107)
at org.apache.spark.storage.DiskBlockObjectWriter.open(DiskBlockObjectWriter.scala:115)
at org.apache.spark.storage.DiskBlockObjectWriter.write(DiskBlockObjectWriter.scala:235)
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:150)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
20/03/28 11:34:04 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[Executor task launch worker for task 212,5,main]
java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedOutputStream.(BufferedOutputStream.java:76)
at org.apache.spark.storage.DiskBlockObjectWriter$ManualCloseBufferedOutputStream$1.(DiskBlockObjectWriter.scala:106)
at org.apache.spark.storage.DiskBlockObjectWriter.initialize(DiskBlockObjectWriter.scala:107)
at org.apache.spark.storage.DiskBlockObjectWriter.open(DiskBlockObjectWriter.scala:115)
at org.apache.spark.storage.DiskBlockObjectWriter.write(DiskBlockObjectWriter.scala:235)
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:150)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
20/03/28 11:34:04 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[Executor task launch worker for task 214,5,main]
java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedOutputStream.(BufferedOutputStream.java:76)
at org.apache.spark.storage.DiskBlockObjectWriter$ManualCloseBufferedOutputStream$1.(DiskBlockObjectWriter.scala:106)
at org.apache.spark.storage.DiskBlockObjectWriter.initialize(DiskBlockObjectWriter.scala:107)
at org.apache.spark.storage.DiskBlockObjectWriter.open(DiskBlockObjectWriter.scala:115)
at org.apache.spark.storage.DiskBlockObjectWriter.write(DiskBlockObjectWriter.scala:235)
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:150)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
20/03/28 11:34:04 WARN TaskSetManager: Lost task 13.0 in stage 84.0 (TID 214,localhost, executor driver): java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedOutputStream.(BufferedOutputStream.java:76)
at org.apache.spark.storage.DiskBlockObjectWriter$ManualCloseBufferedOutputStream$1.(DiskBlockObjectWriter.scala:106)
at org.apache.spark.storage.DiskBlockObjectWriter.initialize(DiskBlockObjectWriter.scala:107)
at org.apache.spark.storage.DiskBlockObjectWriter.open(DiskBlockObjectWriter.scala:115)
at org.apache.spark.storage.DiskBlockObjectWriter.write(DiskBlockObjectWriter.scala:235)
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:150)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

20/03/28 11:34:04 ERROR TaskSetManager: Task 13 in stage 84.0 failed 1 times;aborting job
20/03/28 11:34:04 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerStageCompleted(org.apache.spark.scheduler.StageInfo@78943f6b)
20/03/28 11:34:04 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerJobEnd(9,1585395244749,JobFailed(org.apache.spark.SparkException: Job aborted due to stage failure: Task 13 in stage 84.0 failed 1 times, most recent failure: Lost task 13.0 in stage 84.0 (TID 214, localhost, executor driver): java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedOutputStream.(BufferedOutputStream.java:76)
at org.apache.spark.storage.DiskBlockObjectWriter$ManualCloseBufferedOutputStream$1.(DiskBlockObjectWriter.scala:106)
at org.apache.spark.storage.DiskBlockObjectWriter.initialize(DiskBlockObjectWriter.scala:107)
at org.apache.spark.storage.DiskBlockObjectWriter.open(DiskBlockObjectWriter.scala:115)
at org.apache.spark.storage.DiskBlockObjectWriter.write(DiskBlockObjectWriter.scala:235)
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:150)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Driver stacktrace:))
20/03/28 11:34:04 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerSQLExecutionEnd(3,1585395244753)
20/03/28 11:34:04 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerTaskEnd(84,0,ShuffleMapTask,ExceptionFailure(java.lang.OutOfMemoryError,Java heap space,[Ljava.lang.StackTraceElement;@63b25cb2,java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedOutputStream.(BufferedOutputStream.java:76)
at org.apache.spark.storage.DiskBlockObjectWriter$ManualCloseBufferedOutputStream$1.(DiskBlockObjectWriter.scala:106)
at org.apache.spark.storage.DiskBlockObjectWriter.initialize(DiskBlockObjectWriter.scala:107)
at org.apache.spark.storage.DiskBlockObjectWriter.open(DiskBlockObjectWriter.scala:115)
at org.apache.spark.storage.DiskBlockObjectWriter.write(DiskBlockObjectWriter.scala:235)
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:150)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
,Some(org.apache.spark.ThrowableSerializationWrapper@6caad07d),Vector(AccumulableInfo(5500,Some(internal.metrics.executorRunTime),Some(7810),None,true,true,None), AccumulableInfo(5502,Some(internal.metrics.resultSize),Some(0),None,true,true,None), AccumulableInfo(5503,Some(internal.metrics.jvmGCTime),Some(7329),None,true,true,None), AccumulableInfo(5507,Some(internal.metrics.peakExecutionMemory),Some(2228224),None,true,true,None), AccumulableInfo(5509,Some(internal.metrics.shuffle.read.remoteBlocksFetched),Some(0),None,true,true,None), AccumulableInfo(5510,Some(internal.metrics.shuffle.read.localBlocksFetched),Some(12),None,true,true,None), AccumulableInfo(5511,Some(internal.metrics.shuffle.read.remoteBytesRead),Some(0),None,true,true,None), AccumulableInfo(5512,Some(internal.metrics.shuffle.read.localBytesRead),Some(15733),None,true,true,None),AccumulableInfo(5513,Some(internal.metrics.shuffle.read.fetchWaitTime),Some(0),None,true,true,None), AccumulableInfo(5514,Some(internal.metrics.shuffle.read.recordsRead),Some(1153),None,true,true,None), AccumulableInfo(5516,Some(internal.metrics.shuffle.write.recordsWritten),Some(235),None,true,true,None), AccumulableInfo(5517,Some(internal.metrics.shuffle.write.writeTime),Some(22667687),None,true,true,None)),Vector(LongAccumulator(id: 5500, name: Some(internal.metrics.executorRunTime), value: 7810), LongAccumulator(id: 5502, name: Some(internal.metrics.resultSize), value: 0), LongAccumulator(id: 5503, name: Some(internal.metrics.jvmGCTime), value: 7329), LongAccumulator(id: 5507, name: Some(internal.metrics.peakExecutionMemory), value: 2228224), LongAccumulator(id: 5509, name: Some(internal.metrics.shuffle.read.remoteBlocksFetched), value: 0), LongAccumulator(id: 5510, name: Some(internal.metrics.shuffle.read.localBlocksFetched), value: 12), LongAccumulator(id: 5511, name: Some(internal.metrics.shuffle.read.remoteBytesRead), value: 0), LongAccumulator(id: 5512, name: Some(internal.metrics.shuffle.read.localBytesRead), value: 15733), LongAccumulator(id: 5513, name: Some(internal.metrics.shuffle.read.fetchWaitTime), value: 0), LongAccumulator(id: 5514, name: Some(internal.metrics.shuffle.read.recordsRead), value: 1153), LongAccumulator(id: 5516, name: Some(internal.metrics.shuffle.write.recordsWritten), value: 235), LongAccumulator(id: 5517, name: Some(internal.metrics.shuffle.write.writeTime), value: 22667687))),org.apache.spark.scheduler.TaskInfo@2f7cc6db,org.apache.spark.executor.TaskMetrics@d35d7b9)
20/03/28 11:34:04 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerTaskEnd(84,0,ShuffleMapTask,ExceptionFailure(java.lang.OutOfMemoryError,Java heap space,[Ljava.lang.StackTraceElement;@7c934d61,java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedOutputStream.(BufferedOutputStream.java:76)
at org.apache.spark.storage.DiskBlockObjectWriter$ManualCloseBufferedOutputStream$1.(DiskBlockObjectWriter.scala:106)
at org.apache.spark.storage.DiskBlockObjectWriter.initialize(DiskBlockObjectWriter.scala:107)
at org.apache.spark.storage.DiskBlockObjectWriter.open(DiskBlockObjectWriter.scala:115)
at org.apache.spark.storage.DiskBlockObjectWriter.write(DiskBlockObjectWriter.scala:235)
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:150)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
,Some(org.apache.spark.ThrowableSerializationWrapper@468043b6),Vector(AccumulableInfo(5500,Some(internal.metrics.executorRunTime),Some(7811),None,true,true,None), AccumulableInfo(5502,Some(internal.metrics.resultSize),Some(0),None,true,true,None), AccumulableInfo(5503,Some(internal.metrics.jvmGCTime),Some(7329),None,true,true,None), AccumulableInfo(5507,Some(internal.metrics.peakExecutionMemory),Some(2228224),None,true,true,None), AccumulableInfo(5509,Some(internal.metrics.shuffle.read.remoteBlocksFetched),Some(0),None,true,true,None), AccumulableInfo(5510,Some(internal.metrics.shuffle.read.localBlocksFetched),Some(11),None,true,true,None), AccumulableInfo(5511,Some(internal.metrics.shuffle.read.remoteBytesRead),Some(0),None,true,true,None), AccumulableInfo(5512,Some(internal.metrics.shuffle.read.localBytesRead),Some(16597),None,true,true,None),AccumulableInfo(5513,Some(internal.metrics.shuffle.read.fetchWaitTime),Some(0),None,true,true,None), AccumulableInfo(5514,Some(internal.metrics.shuffle.read.recordsRead),Some(1264),None,true,true,None), AccumulableInfo(5516,Some(internal.metrics.shuffle.write.recordsWritten),Some(123),None,true,true,None), AccumulableInfo(5517,Some(internal.metrics.shuffle.write.writeTime),Some(235120573),None,true,true,None)),Vector(LongAccumulator(id: 5500, name: Some(internal.metrics.executorRunTime), value: 7811), LongAccumulator(id: 5502, name: Some(internal.metrics.resultSize), value: 0), LongAccumulator(id: 5503, name: Some(internal.metrics.jvmGCTime), value: 7329), LongAccumulator(id: 5507, name: Some(internal.metrics.peakExecutionMemory), value: 2228224), LongAccumulator(id: 5509, name: Some(internal.metrics.shuffle.read.remoteBlocksFetched), value: 0),LongAccumulator(id: 5510, name: Some(internal.metrics.shuffle.read.localBlocksFetched), value: 11), LongAccumulator(id: 5511, name: Some(internal.metrics.shuffle.read.remoteBytesRead), value: 0), LongAccumulator(id: 5512, name: Some(internal.metrics.shuffle.read.localBytesRead), value: 16597), LongAccumulator(id: 5513, name: Some(internal.metrics.shuffle.read.fetchWaitTime), value: 0), LongAccumulator(id: 5514, name: Some(internal.metrics.shuffle.read.recordsRead), value: 1264), LongAccumulator(id: 5516, name: Some(internal.metrics.shuffle.write.recordsWritten), value: 123), LongAccumulator(id: 5517, name: Some(internal.metrics.shuffle.write.writeTime), value: 235120573))),org.apache.spark.scheduler.TaskInfo@f030fa3,org.apache.spark.executor.TaskMetrics@2c2899e5)
20/03/28 11:34:04 ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerTaskEnd(84,0,ShuffleMapTask,ExceptionFailure(java.lang.OutOfMemoryError,Java heap space,[Ljava.lang.StackTraceElement;@4ab63144,java.lang.OutOfMemoryError: Java heap space
at java.io.BufferedOutputStream.(BufferedOutputStream.java:76)
at org.apache.spark.storage.DiskBlockObjectWriter$ManualCloseBufferedOutputStream$1.(DiskBlockObjectWriter.scala:106)
at org.apache.spark.storage.DiskBlockObjectWriter.initialize(DiskBlockObjectWriter.scala:107)
at org.apache.spark.storage.DiskBlockObjectWriter.open(DiskBlockObjectWriter.scala:115)
at org.apache.spark.storage.DiskBlockObjectWriter.write(DiskBlockObjectWriter.scala:235)
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:150)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
,Some(org.apache.spark.ThrowableSerializationWrapper@eef4657),Vector(AccumulableInfo(5500,Some(internal.metrics.executorRunTime),Some(7810),None,true,true,None), AccumulableInfo(5502,Some(internal.metrics.resultSize),Some(0),None,true,true,None), AccumulableInfo(5503,Some(internal.metrics.jvmGCTime),Some(7329),None,true,true,None), AccumulableInfo(5507,Some(internal.metrics.peakExecutionMemory),Some(2228224),None,true,true,None), AccumulableInfo(5509,Some(internal.metrics.shuffle.read.remoteBlocksFetched),Some(0),None,true,true,None), AccumulableInfo(5510,Some(internal.metrics.shuffle.read.localBlocksFetched),Some(12),None,true,true,None), AccumulableInfo(5511,Some(internal.metrics.shuffle.read.remoteBytesRead),Some(0),None,true,true,None), AccumulableInfo(5512,Some(internal.metrics.shuffle.read.localBytesRead),Some(10448),None,true,true,None), AccumulableInfo(5513,Some(internal.metrics.shuffle.read.fetchWaitTime),Some(0),None,true,true,None), AccumulableInfo(5514,Some(internal.metrics.shuffle.read.recordsRead),Some(726),None,true,true,None), AccumulableInfo(5516,Some(internal.metrics.shuffle.write.recordsWritten),Some(656),None,true,true,None), AccumulableInfo(5517,Some(internal.metrics.shuffle.write.writeTime),Some(4358799),None,true,true,None)),Vector(LongAccumulator(id: 5500, name: Some(internal.metrics.executorRunTime), value: 7810), LongAccumulator(id: 5502, name: Some(internal.metrics.resultSize), value: 0), LongAccumulator(id: 5503, name: Some(internal.metrics.jvmGCTime), value: 7329), LongAccumulator(id: 5507, name: Some(internal.metrics.peakExecutionMemory), value: 2228224), LongAccumulator(id: 5509,name: Some(internal.metrics.shuffle.read.remoteBlocksFetched), value: 0), LongAccumulator(id: 5510, name: Some(internal.metrics.shuffle.read.localBlocksFetched), value: 12), LongAccumulator(id: 5511, name: Some(internal.metrics.shuffle.read.remoteBytesRead), value: 0), LongAccumulator(id: 5512, name: Some(internal.metrics.shuffle.read.localBytesRead), value: 10448), LongAccumulator(id: 5513, name: Some(internal.metrics.shuffle.read.fetchWaitTime), value: 0), LongAccumulator(id: 5514, name: Some(internal.metrics.shuffle.read.recordsRead), value: 726), LongAccumulator(id: 5516, name: Some(internal.metrics.shuffle.write.recordsWritten), value: 656), LongAccumulator(id: 5517, name: Some(internal.metrics.shuffle.write.writeTime), value: 4358799))),org.apache.spark.scheduler.TaskInfo@19862e3e,org.apache.spark.executor.TaskMetrics@71897b5a)
20/03/28 11:34:04 WARN TaskSetManager: Lost task 7.0 in stage 84.0 (TID 208, localhost, executor driver): TaskKilled (killed intentionally)
20/03/28 11:34:04 WARN TaskSetManager: Lost task 8.0 in stage 84.0 (TID 209, localhost, executor driver): TaskKilled (killed intentionally)
20/03/28 11:34:04 ERROR Executor: Exception in task 3.0 in stage 84.0 (TID 204)
java.nio.file.NoSuchFileException: /tmp/blockmgr-b45c1c38-c730-4191-8c1f-19442a55bb5c/0c/shuffle_16_3_0.index.4bc33655-9aef-43ae-b116-b96c14295075
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434)
at java.nio.file.Files.newOutputStream(Files.java:216)
at org.apache.spark.shuffle.IndexShuffleBlockResolver.writeIndexFileAndCommit(IndexShuffleBlockResolver.scala:146)
at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:163)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
20/03/28 11:34:04 ERROR TaskContextImpl: Error in TaskCompletionListener
java.lang.IllegalStateException: Block broadcast_27 not found
at org.apache.spark.storage.BlockInfoManager$$anonfun$2.apply(BlockInfoManager.scala:293)
at org.apache.spark.storage.BlockInfoManager$$anonfun$2.apply(BlockInfoManager.scala:293)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.storage.BlockInfoManager.unlock(BlockInfoManager.scala:292)
at org.apache.spark.storage.BlockManager.releaseLock(BlockManager.scala:671)
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$releaseLock$1.apply(TorrentBroadcast.scala:246)
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$releaseLock$1.apply(TorrentBroadcast.scala:246)
at org.apache.spark.TaskContext$$anon$1.onTaskCompletion(TaskContext.scala:123)
at org.apache.spark.TaskContextImpl$$anonfun$markTaskCompleted$1.apply(TaskContextImpl.scala:97)
at org.apache.spark.TaskContextImpl$$anonfun$markTaskCompleted$1.apply(TaskContextImpl.scala:95)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at org.apache.spark.TaskContextImpl.markTaskCompleted(TaskContextImpl.scala:95)
at org.apache.spark.scheduler.Task.run(Task.scala:112)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
20/03/28 11:34:04 ERROR Executor: Exception in task 16.0 in stage 84.0 (TID 217)
java.util.NoSuchElementException: None.get
at scala.None$.get(Option.scala:347)
at scala.None$.get(Option.scala:345)
at org.apache.spark.storage.BlockInfoManager.releaseAllLocksForTask(BlockInfoManager.scala:348)
at org.apache.spark.storage.BlockManager.releaseAllLocksForTask(BlockManager.scala:687)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:329)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
20/03/28 11:34:04 WARN NettyRpcEnv: RpcEnv already stopped.
[anubhav68649161@cxln4 ~]$