[prassadht2483@cxln5 ~]$ spark-submit arpincan.py
SPARK_MAJOR_VERSION is set to 2, using Spark2
Traceback (most recent call last):
File “/home/prassadht2483/arpincan.py”, line 4, in
from commons.Utils import Utils
ImportError: No module named commons.Utils
[prassadht2483@cxln5 ~]$ nano arpincan.py
[prassadht2483@cxln5 ~]$ spark-submit arpincan.py
SPARK_MAJOR_VERSION is set to 2, using Spark2
Traceback (most recent call last):
File “/home/prassadht2483/arpincan.py”, line 4, in
from commons.Utils import Utils
ImportError: No module named commons.Utils
[prassadht2483@cxln5 ~]$ cat arpincan.py
import sys
sys.path.insert(0, ‘.’)
from pyspark import SparkContext, SparkConf
from commons.Utils import Utils
def splitComma(line: str):
splits = Utils.COMMA_DELIMITER.split(line)
return “{}, {}”.format(splits[1], splits[2])
if name == “main”:
conf = SparkConf().setAppName(“airports”).setMaster(“local[*]”)
sc = SparkContext(conf = conf)
airports = sc.textFile(“untitled36.txt”)
airportsInCanada = airports.filter(lambda line : Utils.COMMA_DELIMITER.split(line)[3] == ““Canada””)
airportsNameAndCityNames = airportsInCanada.map(splitComma)
airportsNameAndCityNames.saveAsTextFile(“output_Canada”)