Spark-shell command is throwing error

Lab Support
#1

[arunhdp7619@ip-172-31-38-146 ~]$ spark-shell
SPARK_MAJOR_VERSION is set to 2, using Spark2
File “/bin/hdp-select”, line 242
print "ERROR: Invalid package - " + name
^
SyntaxError: Missing parentheses in call to ‘print’. Did you mean print("ERROR: Invalid package - " + name)?
ls: cannot access /usr/hdp//hadoop/lib: No such file or directory
Exception in thread “main” java.lang.IllegalStateException: hdp.version is not set while running Spark under HDP, please set through HDP_VERSION in spark-env.sh or add a ja
va-opts file in conf with -Dhdp.version=xxx
at org.apache.spark.launcher.Main.main(Main.java:118)

spark-shell%20error
spark-shell%20error1440×900 151 KB

Please help me to resolve this issue.

Thanks,
Arun

#2

Hi,

I am also facing the same error. Please help me to sort out this

Thanks & Regards,
Shajahan Anwarbasha

#3

Use below command first:
export SPARK_MAJOR_VERSION=1
then
spark-shell

#4

Hi all,

This error is resolved. Please check after login again.

Also, on web console of Cloudxlab, the ‘python’ command, by default, runs ‘Python version 2’. To run ‘Python version 3’ on the web console, you can run the below command after the login.

export PATH=/usr/local/anaconda/bin:$PATH

Thanks and regards,

Deepak

1 Like
#5

Hi Deepak,

Thanks for the reply. It is working now…

–Arun.

#6

export HDP_VERSION=xxx worked for me.