rdd = sc.textFile("/user/kkgprest27288/myfirstfile.txt")
DF = rdd.flatMap(lambda x:x.split(" ")).map(lambda x:(x,1)).reduceByKey(lambda x, y:x+y).toDF([“word”,“count”])
I ran my word count program and I am getting following error message:
20/06/12 06:45:43 WARN TaskSetManager: Lost task 1.0 in stage 87.0 (TID 121, cxln3.c.thelab-240901.internal, executor 1): java.io.IOException: Failed to create local dir in /hadoop/yarn/local/userc
ache/kkgprest27288/appcache/application_1588872731573_13657/blockmgr-bf27b6f0-c37f-4c30-ba57-9a7c539076a4/34.
could you please help me out?