Hi Team,
I want to run spark submit command on webconsole . But I am getting the error: “user ** is not allowed to execute '/bin/spark-submit as root on cxln4”. I am using free 7 days lab . Is the permission denied for this subscription or I need enable something .
Can you paste the command?
Hi Abhinav,
I want to know first,
is it possible to run spark scala jar file, uploaded to HDFS using Hue, by spark submit command in this lab?
You can do all these things
I want to execute the labtest.jar file which is available in HDFS.
I am getting the following error for spark submit. Is this because the permission issue for accessing jar file?
When I tried same from spark query editor, I got the below message.
You can use Spark only on web console
Also on console the error is self-explanatory
why it is showing jar does not exist? Is there any problem on my spark submit command?
Hope the class not found exception is coming because it is not detecting the jar file in HDFS.
labtest.jar contain the below code:
package com.lab.test
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.rdd.RDD.rddToPairRDDFunctions
object labtest {
def main(args: Array[String]) = {
val conf = new SparkConf().setAppName(“WordCount”).setMaster(“local”)
val sc = new SparkContext(conf)
val input = sc.textFile(“inputforsparkwordcountapp.txt”)
input.flatMap { line => line.split(" ")
} //for each line split the line into words.
.map { word => (word, 1)}
.reduceByKey(_ + _)
.saveAsTextFile(“outputfiletest”)
sc.stop()
}
}
The jar file must be in the local file system not in HDFS.
use hadoop fs -get filename to copy it locally.
Thanks sgiri.
I copied files to local and run the submit matser as local. now it’s working…
1 Like