Unable to create spark dataframe in JUpyter notes

Lab Support
#1

I’m executing the following commands

data = [
(1, ‘a’),
(2, ‘b’),
(3, ‘c’),
(4, ‘d’),
(5, ‘e’),
(6, ‘a’),
(7, ‘b’),
(8, ‘c’),
(9, ‘d’),
(10, ‘e’)
]

df = sqlContext.createDataFrame(data, [‘numbers’, ‘letters’])

But dataframe is not getting create eventhough I’ve given all necessary permissions on directory:

Caused by: org.apache.hadoop.security.AccessControlException: Permission denied: user=ashokreddypalepalli5966, access=WRITE, inode="/home/ashokreddypalepalli5966/ashok_spark/spark-warehouse":hdfs:hdfs:drwxr-xr-x

#2

@ashok_reddy_Palepall

Instead of /home/, can you try /user/, as ‘home’ refers to your linux home directory, where as ‘user’ refers to HDFS.

1 Like