Spark lab support- technical support

Lab Support
#1

how to start spark shell? I have run below command but got error:
[knsujith5240@ip-172-31-20-58 ~]$ spark shell
-bash: spark: command not found

#2

Hi @Sujith_K

You command is wrong. Correct command is

spark-shell

Hope this helps

#3

Thanks Abhinav.

I am trying to create spark context but it is getting failed with below error:

org.apache.spark.SparkException: Only one SparkContext may be running in this JVM (see SPARK-2243).
To ignore this error, set spark.driver.allowMultipleContexts = true

i was running below code :

// Create a local StreamingContext with two working thread and batch interval of 1 second.

val conf = new SparkConf().setMaster(“local[2]”).setAppName(“NetworkWordCount”)
val ssc = new StreamingContext(conf, Seconds(5))

// Create a DStream that will connect to hostname:port, like localhost:9999
val lines = ssc.socketTextStream(“localhost”, 9999)

#4

Hi @Sujith_K,

Which Spark version are you using and can you please let me know your project directory so that I can checkout the code?

Thanks

#5

Hi @abhinav

Spark version 1.5.2

Below code i am trying to execute in spark-shell

====
import org.apache.spark._
import org.apache.spark.streaming._
import org.apache.spark.streaming.StreamingContext._

// Create a local StreamingContext with two working thread and batch interval of 5 second.

val conf = new SparkConf().setMaster(“local[2]”).setAppName(“NetworkWordCount”)
val ssc = new StreamingContext(conf, Seconds(5))

// Create a DStream that will connect to hostname:port, like localhost:9999
val lines = ssc.socketTextStream(“localhost”, 9999)

// Split each line into words
val words = lines.flatMap(_.split(" "))

// Count each word in each batch
val pairs = words.map(word => (word, 1))
val wordCounts = pairs.reduceByKey(_ + _)

wordCounts.print()

ssc.start() // Start the computation

#6

Hi @Sujith_K,

If you are running this code on spark-shell then you should not create Spark Context again as Spark shell already creates a one spark context for you in sc variable

Hope this helps.

Thanks

#7

Hi Abhinav,

But in the above code he is creating SSC (streaming Context). I am also getting the same error as above. What change should be done in above code?