Unable to access SparkSession in Spark 2.3 after upgrade

Lab Support
#1

Hi @abhinavsingh & Team,

I am unable to run my Jupyter Notebook which was previously working in Spark 2.x after the lab upgrade. May i know how to fix this?

image
image1173×573 39.5 KB

#2

Hey @senthiljdpm,

Is this still coming?

#3

it works now… thank you

  1. May i know how do i submit pyspark job using spark-submit in the shell?
  2. Should my pyspark script include the following lines while submitting the script to spark-submit?

import os
import sys
os.environ[“SPARK_HOME”] = “/usr/hdp/current/spark2-client”
os.environ[“PYLIB”] = os.environ[“SPARK_HOME”] + “/python/lib”
os.environ[“PYSPARK_PYTHON”] = “/usr/local/anaconda/bin/python”
os.environ[“PYSPARK_DRIVER_PYTHON”] = “/usr/local/anaconda/bin/python”
sys.path.insert(0, os.environ[“PYLIB”] +"/py4j-0.10.6-src.zip")
sys.path.insert(0, os.environ[“PYLIB”] +"/pyspark.zip")

#4

Please find my answers below

May i know how do i submit pyspark job using spark-submit in the shell?

Please check Spark documentation for the same.

Should my pyspark script include the following lines while submitting the script to spark-submit?

If you are running spark-submit from the command line then you do not require the above Python code.

Hope this helps.

Thanks