Pyspark spark submit not working

Technical Discussions Python
#1

HI Team ,
I have loaded one of my sample [NASAhosts.py] loaded alomg
the path :slight_smile: (https://jupyter.f.cloudxlab.com/user/asmitaece887002/edit/NASAhosts.py)I am trying to submit through pyspark using the following command and getting the below error . —
[asmitaece887002@cxln5 ~]$ spark-submit NASAhosts.py
SPARK_MAJOR_VERSION is set to 2, using Spark2
File “/bin/hdp-select”, line 232
print "ERROR: Invalid package - " + name
^
SyntaxError: Missing parentheses in call to ‘print’. Did you mean print("ERROR: Invalid package - " + name)?
ls: cannot access /usr/hdp//hadoop/lib: No such file or directory
Exception in thread “main” java.lang.IllegalStateException: hdp.version is not set while running Spark under HDP, please set through HDP_VERSION in spark-env.sh or add a java-opts file
in conf with -Dhdp.version=xxx
at org.apache.spark.launcher.Main.main(Main.java:118)
=-----------------------------------------

even tried this :slight_smile:spark-submit --py-NASAhosts.py
SPARK_MAJOR_VERSION is set to 2, using Spark2
File “/bin/hdp-select”, line 232
print "ERROR: Invalid package - " + name
^
SyntaxError: Missing parentheses in call to ‘print’. Did you mean print("ERROR: Invalid package - " + name)?
ls: cannot access /usr/hdp//hadoop/lib: No such file or directory
Error: Unrecognized option: --py-NASAhosts.py
Exception in thread “main” java.lang.IllegalStateException: hdp.version is not set while running Spark under HDP, please set through HDP_VERSION in spark-env.sh or add a java-opts file
in conf with -Dhdp.version=xxx
at org.apache.spark.launcher.Main.main(Main.java:118)

Please help on this

#2

HI, Asmita.

Did you try giving the no of worker nodes which will be processing your commands?

For example:- spark-submit --master “local[2]” NASAhosts.py
This will run spark locally with two worker threads as logical cores on your machine.

All the best!

#3

Hi satyajit , i did not .but somehow googling , i found that this error of hdp comes because hdp is compatible to python 2 and not python 3 …on nit setting the python path to 3 , this code runs with spark submit .but on setting the path to python 3 , this code doesn’t .can you please help on this ?