Spark UI on CloudxLab


#1

I am running a spark-shell using Spark2 available in CloudxLab however I am unable to access Spark UI.
Request you to please let me know in which port Spark UI can be accessed in CloudxLab ?


Spark Master URL & Port
#2

I Think this Port is 3002
But Please check How to Access Spark 2.0

for Access Spark 2.0:

1- Login to Web Console
2-Set the environment variables
3-So that spark can discover hadoop

export YARN_CONF_DIR=/etc/hadoop/conf/
export HADOOP_CONF_DIR=/etc/hadoop/conf/

4- Launch scala spark shell
/usr/spark2.0.1/bin/spark-shell

5- Create RDD from text file in HDFS
var r = sc.textFile("/data/mr/wordcount/input");

6- It should display first 10 lines of the file
r.take(10)

7- Launch PySpark
/usr/spark2.0.1/bin/pyspark


#3

Thanks for the response. However using the instructions provided I was not able to bring up the Spark UI in the browser.Do I need to use the complete IP plus the port to bring the sparkUI ? Also I submitted a spark streaming job but I was not able to find any log or any thing related to Spark UI or Spark History Server.is there a way to find the link of these two (UI and History server).I tried to search in Resouce Manager UI there also no luck.


Cannot connect to application master from Resource Manager - Spark UI
#4

Hi Deepak,

Let’s say your web console in on e.cloudxlab.com and your Spark job is running on port 4045.
To access Spark UI, you will have to type in http://e.cloudxlab.com:4045 in your browser

Also, Spark History Server for Spark 1.6 is located at http://c.cloudxlab.com:18080/ and for Spark 2.3 is located at http://c.cloudxlab.com:18081/

Hope this helps.

Thanks


I'm not able to open Spark UI
Accessing Spark 2.2.1 web UI
Give me the path for local and hdfs seperately
CaNT SEE MY USER NAME AND JIB RUN IN SPARK SHELL IN sparkui
How to open the url to track the job
#5

Please find port number in the console from where you are running Spark job


#6

the above steps to access Spark UI is not working, can you check this out with the latest steps…


#7

Hi @Jeevankrishna_Rao,

I have updated my answer to include new URLs and ports.

By the way, you can find the spark history server URLs in Ambari also.

Hope this helps.

Thanks


#8

Hi @abhinav,
I tried what you have suggested. But unable to open spark web UI. Please help.

My spark shell started at 4045 port. and I am working on https://e.cloudxlab.com/. So i tried with http://e.cloudxlab.com:4045/...error message is unable to connect.

[vinpaluvuri5873@cxln4 ~]$ spark-shell
SPARK_MAJOR_VERSION is set to 2, using Spark2
Setting default log level to “WARN”.
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
19/08/02 03:02:07 WARN Utils: Service ‘SparkUI’ could not bind on port 4040. Attempting port 4041.
19/08/02 03:02:07 WARN Utils: Service ‘SparkUI’ could not bind on port 4041. Attempting port 4042.
19/08/02 03:02:07 WARN Utils: Service ‘SparkUI’ could not bind on port 4042. Attempting port 4043.
19/08/02 03:02:07 WARN Utils: Service ‘SparkUI’ could not bind on port 4043. Attempting port 4044.
19/08/02 03:02:07 WARN Utils: Service ‘SparkUI’ could not bind on port 4044. Attempting port 4045.
Spark context Web UI available at http://10.142.1.4:4045

Regards,
Vinod.


#9

Hi,
I am able to open the spark UI.

Please note that the IP address displayed is privated. From “My Lab” I found the corresponding public name which e.cloudxlab.com.

Then, I open the UI using the public name: