I have set my PYSPARK variables as follows.
os.environ[“PYSPARK_PYTHON”] = “/usr/bin/python2.7”
os.environ[“PYSPARK_DRIVER_PYTHON”] = “/usr/bin/python2.7”
I keep getting the following error.
Exception: Python in worker has different version 2.7 than that in driver 3.6, PySpark cannot run with different minor versions.Please check environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set.