Spark-submit cluster mode - NoClassDefFoundError

Technical Discussions
#1

i am getting below error while submitting spark job in cluster mode.
–deploy-mode client model is working fine
/usr/spark2.0.2/bin/ spark-submit --name hello --master yarn --deploy-mode client --executor-memory 1g --executor-cores 1 --num-executors 1 --class com.first.load.Hello /home/udaychitukula6587/first.load/target/scala-2.10/helloworld_2.10-1.0.jar

Below one throwing error:
/usr/spark2.0.2/bin/ spark-submit --name hello --master yarn --deploy-mode cluster --executor-memory 1g --executor-cores 1 --num-executors 1 --class com.first.load.Hello /home/udaychitukula6587/first.load/target/scala-2.10/helloworld_2.10-1.0.jar
ERROR:
:: problems summary ::
:::: ERRORS
_ unknown resolver null_

:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
:: retrieving :: org.apache.spark#spark-submit-parent
_ confs: [default]_
_ 0 artifacts copied, 17 already retrieved (0kB/19ms)_
Exception in thread “main” java.lang.NoClassDefFoundError: com/sun/jersey/api/client/config/ClientConfig
_ at org.apache.hadoop.yarn.client.api.TimelineClient.createTimelineClient(TimelineClient.java:55)_
_ at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createTimelineClient(YarnClientImpl.java:181)_
_ at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:168)_

this program is to print welcome message

please help me in resolving this
thanks in advance

#2

could someone please look into the issue???

#3

Hi @Uday_Reddy,

Spark 2.x will not work in YARN mode currently. It is a pending issue which will solve during the cluster upgrade.

We will let you know as soon as we will upgrade the cluster. Extremely sorry for the inconvenience. Please run Spark 2.x in local mode for now.

Thanks

#4

Hi,
Is still this issue exists, I am getting same issue.

#5

I have tested the yarn mode multiple times. It seems to work. Please share the code.

#6

Issue still persists.

I ran below command on web-console:

/usr/spark2.4.3/bin/spark-submit --master yarn --class com.kaggle.lendingclubloans.LoanRiskClassifier lending-club-loans-application_2.11-1.0.jar loan2.csv

It failed due to below error:
Exception in thread "main" java.lang.NoClassDefFoundError: com/sun/jersey/api/client/config/ClientConfig

I found that com.sun.jersey.api.client.config.ClientConfig is available only till jersey 1.9 library. Spark 2.4.3 on Cloudxlab has jersey 2.22 libraries. Jersey 1.9 libraries need to be added to below jars directory.

[umeshdeorukhkar9526@cxln5 jars]$ pwd

/usr/spark2.4.3/jars

[umeshdeorukhkar9526@cxln5 jars]$ ls *jersey*

jersey-client-2.22.2.jar jersey-container-servlet-2.22.2.jar jersey-guava-2.22.2.jar jersey-server-2.22.2.jar

jersey-common-2.22.2.jar jersey-container-servlet-core-2.22.2.jar jersey-media-jaxb-2.22.2.jar

[umeshdeorukhkar9526@cxln5 jars]$