Pyspark not working in yarn mode

Lab Support
#1

Hello team,

I am trying to open pysaprk with yarn mode, but I am getting the below error. Please advise.

pyspark – master yarn -conf spark.ui.port=12456 --num-executors 20
Exception in thread “main” java.lang.IllegalArgumentException: pyspark does not support any application options.
at org.apache.spark.launcher.CommandBuilderUtils.checkArgument(CommandBuilderUtils.java:254)
at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildPySparkShellCommand(SparkSubmitCommandBuilder.java:243)
at org.apache.spark.launcher.SparkSubmitCommandBuilder.buildCommand(SparkSubmitCommandBuilder.java:117)
at org.apache.spark.launcher.Main.main(Main.java:87)

#2

Hi @Kranthi_Karnam,

The options are not properly passed. Could you try with below command?

pyspark --master yarn

Hope this helps.

Thanks

1 Like
#3

Thanks abhinav,it worked.

1 Like