Spark-submit class not found exception

Technical Discussions Apache Spark
#1

Hi Team,

I am trying to run this simple spark application with the spark submit command.

package org.apache.spark.SparkProgram

/**

  • Created by Maheshn on 8/17/2017.
    */
    import org.apache.spark.SparkConf
    import org.apache.spark.SparkContext
    import org.apache.spark.SparkContext._

object SparkTest {
def main(args: Array[String]): Unit = {

    val conf = new SparkConf().setAppName("count")
    val sc = new SparkContext(conf)
    val list = List("Hello world", "Hello Mahesh howe are you", "Hello s", "sql context is available as sqlcontext")
    val parallel = sc.parallelize(list)
    val record = parallel.flatMap(rec => rec.split(" ")).map(rec => (rec, 1)).reduceByKey(_ + _).map(rec => rec.productIterator.mkString("\t"))
    record.collect().foreach(println)

}

}

When i try to run it with below command

spark-submit --class " SparkTest" --master local[4] sparkdemo_2.11-1.0.jar

I encounter classNotFoundException .

Please help .Thank you

#2

Resolved …

it was an issue in giving below command

spark-submit --class " SparkTest" --master local[4] sparkdemo_2.11-1.0.jar

Instead of above i gave command

spark-submit --class org.apache.spark.SparkProgram.SparkTest --master yarn /home/user/sparkdemo_2.11-1.0.jar

2 Likes
#3

Thank Mahesh for posting the answer here so that others could also use it.

For others, you need to specify the fully qualified name in -class argument.

#4

Hi Mahesh/Sandip Sir,

Please help on the below issue while executing the spark application using spark-submit.
spark-submit --class org.apache.spark.project.Demo2 --master yarn /home/mdakshita01218889/SparkApp1-0.0.1-SNAPSHOT.jar

error

21/08/17 13:42:59 INFO EventLoggingListener: Logging events to hdfs:///spark2-history/local-1629207777822
21/08/17 13:43:03 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.io.IOException: unexpected exception type

java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: scala/runtime/LambdaDeserialize
at org.apache.spark.project.Demo2$.$deserializeLambda$(Demo2.scala)

#5

in my laptop scala version is 2.13, spark dependent packages are as below in pom.xml file

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.12</artifactId>
    <version>2.4.0</version>
</dependency>

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql_2.12</artifactId>
    <version>2.4.0</version>
    <scope>provided</scope>
</dependency>
#6

Please help on the above. I have struggling with this issue since so many days.
Below is my spark application program it is running well in eclipse ide.
package org.apache.spark.project
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._

object Demo2{
case class Emp(id:Int,name:String)
def main(args:Array[String]){
val spark = SparkSession.builder().master(“local”).appName(“exampleOfUdf”).getOrCreate

spark.sparkContext.setLogLevel("error")

import spark.implicits._
val l1 = List((101,"madhu"),(102,"sruthi"))
val rdd1 = spark.sparkContext.parallelize(l1)
val df1 = rdd1.map(x=>Emp(x._1,x._2))
val ds1 = df1.toDS()
ds1.show

val ds2 = ds1.map(x=>{
       val list1 = List(x.id,x.name)
       list1.mkString
})

rdd1.map(x=>{
  val list1 = List(x._1,x._2)
       list1.mkString
}).foreach(println)

ds2.show


spark.close()

}
}