Error while looking for metadata directory

#1

textdata = spark.read.text("/user/kkgprest27288/myfirstfile.txt")
when I tried to run above command. I am getting following error message
20/06/12 07:10:53 WARN DataSource: Error while looking for metadata directory.

#2

Hi @kkraj,

We have provided the instructions in the video to access hive in Spark. In short, you have to provide hive.xml environment variable to Spark. Can you please follow those configurations as suggested in the course video.

#3

Abhinav,

Which video are you referring ? Please let me know video link

#4

@abhinav , @Shubh_Tripathi , @sandeepgiri - I am facing the same issue. Could you please provide the link to the video or instructions in this message chain.

#5

Sanchit,

Could you share the detailed steps to reproduce this error?

#6

launch pyspark shell then below code -

df_orders = spark.readStream.format(“csv”)
.option(“path”,“input_folder”)
.option(“header”,“true”)
.option(“inferSchema”,“true”)
.load()

df_orders.createOrReplaceTempView(“df_orders_file”)

df_all_orders = spark.sql(“select * from df_orders_file”)

order_stream = df_all_orders.writeStream
.format(“csv”)
.outputMode(“append”)
.option(“checkpointLocation”,“checkpoint-location10”)
.option(“path”,“output_folder”)
.start()

#7

Any update here @sandeepgiri ?

#8

Any update @sandeepgiri @Shubh_Tripathi

#9

Hi Sanchit,

try this:

df_orders = spark.read.format('csv').option('path','input_folder').option('header','true').option('inferSchema','true').load()

Also, is there a particular reason of using readStream?
Note:- You can get more knowledge about these concepts at My courses