Dataset API not working

#1

Hi,

I am trying to execute below command to create a Dataset API. but it is throwing an error. Could you please check and update me.

scala> val ds = Seq(1, 2, 3).toDS()
:25: error: value toDS is not a member of Seq[Int]
val ds = Seq(1, 2, 3).toDS()
^

scala> case class Person(name:String, age:Long)
defined class Person
scala> val ds = Seq(Person(“amma”,10),Person(“avva”,15)).toDS()
:27: error: value toDS is not a member of Seq[Person]
val ds = Seq(Person(“amma”,10),Person(“avva”,15)).toDS()
^

Thanks,
Mahesh .H

#2

Hi @maheshharuvaiah8134,

I think you are running these commands on scala shell and not on spark-shell. Please run above code in spark-shell and it should just work fine.

Hope this helps.

Thanks

#3

This is the command which i am executing to enter into spark shell. Please confirm whether it is correct…

[maheshharuvaiah8134@ip-172-31-60-179 buchipalli]$ spark-shell

Thanks,
Mahesh .H

#4

Can you please let me know which Spark version are you using?

#5

Spark - version 1.5.2
Scala version 2.10.4

If this is not the right one, please let me know which one to use and how to use…

Thanks,
Mahesh .H

#6

And also, please explain me what does the below command mean…

[maheshharuvaiah8134@ip-172-31-60-179 sreeram]$ spark-sql

Thanks,
Mahesh .H

#7

@maheshharuvaiah8134

The dataset API is not part of Spark 1.5.2. Please use Spark 2.0.2 for the same.

Refer this blog to access various Spark versions on CloudxLab.

Hope this helps.

Thanks

#8

Thanks for letting me know… I have started using spark2.0.1…

But i am facing lot of issues while using the 2.0.1 wherein i was able to perform below command s successfully on 1.5.2. Please guide me…

scala> val data = sc.textFile(“file:///home/maheshharuvaiah8134/scala_j/sreeram/emp_info”)

scala> case class EMP(id:Int, name:String, sal:Int, desig:String, dname:String)

scala> val data1 = data.map(x => x.split(","))

scala> val empd = data1.map(x => EMP(x(0).toInt, x(1).trim, x(2).toInt, x(3).trim, x(4).trim))

scala> val empDF = empd.toDF

Getting the below error::::

Caused by: org.apache.derby.iapi.error.StandardException: Failed to start database ‘metastore_db’ with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@36b5dbdf, see the next exception for details.
at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
at org.apache.derby.impl.jdbc.SQLExceptionFactory.wrapArgsForTransportAcrossDRDA(Unknown Source)
… 140 more
Caused by: org.apache.derby.iapi.error.StandardException: Another instance of Derby may have already booted the database /home/maheshharuva
iah8134/scala_j/ramsri_1/metastore_db.

Thanks,
Mahesh .H

#9

You would need to import implicits in the following way: import spark.implicits._

#10

Hi,

I will check and get back to you. Thanks.

Thanks,
Mahesh .H

#11

Did it work after you imported the implicits?