Input path does not exist

Technical Discussions
#1

Hi Team ,
My text file is present in both location .
HDFS and local .
i have mentioned correct path as well but couldnt complete the below command .
please help
please find below code ::

scala> var RDD = sc.textFile("/home/aniketptl3487234/myfirstfile.txt")
RDD: org.apache.spark.rdd.RDD[String] = /home/aniketptl3487234/myfirstfile.txt MapPartitionsRDD[1] at textFile at :24
scala> RDD.take(2)
org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://cxln1.c.thelab-240901.internal:8020/home/aniketptl3487234/myfirstfi
le.txt
at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:287)
at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:229)
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:315)
at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:202)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)
at org.apache.spark.rdd.RDD$$anonfun$take$1.apply(RDD.scala:1333)
at org.apache.spark.rdd.RDDOperationScope$.withSco

#2

Hi, please try the following path instead:

~/myfirstfile.txt

#3

Hi Raj Sir, @rajtilakb

i tried as you mentioned but No luck .

Can please help
PLease find below code

scala> var linesRDD=sc.textFile("~/myfirstfile.txt")
linesRDD: org.apache.spark.rdd.RDD[String] = ~/myfirstfile.txt MapPartitionsRDD[5] at textFile at :24
scala> linesRDD.take(10)
org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://cxln1.c.thelab-240901.internal:8020/user/aniketptl3487234/~/myfirst
file.txt
at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:287)
at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:229)
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:315)
at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:202)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)
at org.apache.spark.rdd.RDD$$anonfun$take$1.apply(RDD.scala:1333)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
at org.apache.spark.rdd.RDD.take(RDD.scala:1327)
… 48 elided

#4

Hi,

Please check the error message. It says:

Tilda is definitely not supported. Please try these if ‘myfirstfile.txt’ is existing in your home directory of HDFS.

var RDD1 = sc.textFile("myfirstfile.txt")
RDD1.take(2)

var RDD2 = sc.textFile("/user/aniketptl3487234/myfirstfile.txt")
RDD2.take(2)
1 Like
#5

@sandeepgiri
Hi Sandeep Sir…
i have just started learning the bigdata .
and some hands-on
I was aware this is very basic question but i couldnt figured out.
Thank you so much
Its working now.
Im in love with cloudx lab !!!

Once again thank you for response

1 Like