Spark master node addrees with port

Lab Support
#1

I wanted to connect to spark2 for spark streaming purpose from my java code.

This code I’m using when I’m running spark on my local machine.
SparkConf configuration = newSparkConf().setAppName(“kafka-consumer”).setMaster(“local[*]”);
JavaSparkContext sparkContext = new JavaSparkContext(configuration);

But my question is that when I want to connect to remote spark that is available in cloudxlab, then what addrees, port or connection string I need to use inside the setMaster.

Please let me know ASAP.

Thanks

#2

Hi,

You may not be able to connect to YARN or spark from remote.

Instead, use: .setMaster(“yarn”)

Afterwards, upload your code to cloudxlab home and compile it there using sbt. Once done, you can run using spark-submit.

Please see here: https://cloudxlab.com/assessment/playlist-intro/18/writing-spark-applications?course_id=1&playlist_id=18