Please help me how we can check if the spark-sql is running in Apache Toree in Jupyter.
My command is :
val batsmandf = spark
.read
.format(“csv”)
.option(“inferSchema”, “true”)
.option(“header”, “false”)
.load("/home/isusmitabiswas3880/Batting_data.csv")
// COMMAND ----------
batsmandf.show
The output I got
Name: org.apache.spark.sql.AnalysisException
Message: Path does not exist: hdfs://cxln1.c.thelab-240901.internal:8020/home/isusmitabiswas3880/Batting_data.csv;
StackTrace: at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$14.apply(DataSource.scala:382)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$14.apply(DataSource.scala:370)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.immutable.List.foreach(List.scala:381)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
at scala.collection.immutable.List.flatMap(List.scala:344)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:370)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:152)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:135)