Cannot load main class from JAR file

Technical Discussions Apache Spark
#1

I am trying to execute “spark-submit labtest.jar --master yarn --class com.lab.test.labtest”. Getting error " Cannot load main class from JAR file:/home/aswiniharidas194641/labtest.jar".

Please help me solve this error. jar file uploaded in hue.

My jar file contains below code:
package com.lab.test
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.rdd.RDD.rddToPairRDDFunctions

object labtest {
def main(args: Array[String]) = {

val conf = new SparkConf().setAppName(“WordCount”).setMaster(“local”)
val sc = new SparkContext(conf)
val input = sc.textFile(“inputforsparkwordcountapp.txt”)
input.flatMap { line => line.split(" ")
} //for each line split the line into words.
.map { word => (word, 1)}
.reduceByKey(_ + _)
.saveAsTextFile(“outputfiletest”)
sc.stop()
}

}

Thanks,
Aswini

#2

Hi, Aswini.

I understand that this is your own test .jar file that you have uploaded into your hdfs and now submitting to the spark for word count.

This is the basic syntax for this.
spark-submit --class “WordCount” --master “local[2]” target/scala-2.10/word-count_2.10-1.0.jar
I see that your class name is labtest, but you given labtest.jar in class name.
Can you check it once?

All the best.

#3

Thanks for the reply Satyajit.

spark-submit labtest.jar --master yarn --class com.lab.test.labtest -> this is the spark submit command i ran.
class name:com.lab.test.labtest
jar file name:labtest.jar.

Jar file uploaded to hdfs using hue.

#4

@team ,

I am running simple program using spark submit command , getting below error ,
spark-submit --master yarn --deploy-mode cluster create_spark_session
could you please help me out.
Regards,
Ganesh