SPARK foldLeft /reduceLeft/ rightnot working and fold giving different result

Technical Discussions
#1

Hi
I am getting following error while trying to use foldLeft/foldRight/reduceLeft/reduceRight in spark but when using same thing in my local scala console working fine.
Error:

scala> dataRDD.foldLeft(1)(_ + )<console>:26: error: value foldLeft is not a member of org.apache.spark.rdd.RDD[Int] dataRDD.foldLeft(1)( + _)

dataRDD.reduceLeft(_ + )
:26: error: value reduceLeft is not a member of org.apache.spark.rdd.RDD[Int]
dataRDD.reduceLeft( + _)

Also when using only fold function with initial value 1 for list mentioned below I think I am getting wrong answer coming 35 instead of 27 …could you pleas check ?

scala> val data = List(1,2,3,5,7,8)data: List[Int] = List(1, 2, 3, 5, 7, 8)

scala> dataRDD.fold(1)(_ + _) res7: Int = 35

#2

Can anybody reply me please? Why these function not working in spark/scala in cloudexLab.

Thanks

#3

dataRDD - is a spark RDD ( org.apache.spark.rdd.RDD). It’s api does not have any foldLeft and reduceLeft Method. Please see the API here: https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.rdd.RDD

image
image1102×354 42.9 KB

The foldLeft, foldRight methods are only the members of scala List class:

image
image1192×187 28.2 KB

See the documentation here: https://www.scala-lang.org/api/current/scala/collection/immutable/List.html

1 Like