Spark UI on CloudxLab

[Deepak_Sharma]

I am running a spark-shell using Spark2 available in CloudxLab however I am unable to access Spark UI.
Request you to please let me know in which port Spark UI can be accessed in CloudxLab ?

[n.karimiazar]

I Think this Port is 3002
But Please check How to Access Spark 2.0

for Access Spark 2.0:

1- Login to Web Console
2-Set the environment variables
3-So that spark can discover hadoop

export YARN_CONF_DIR=/etc/hadoop/conf/
export HADOOP_CONF_DIR=/etc/hadoop/conf/

4- Launch scala spark shell
/usr/spark2.0.1/bin/spark-shell

5- Create RDD from text file in HDFS
var r = sc.textFile("/data/mr/wordcount/input");

6- It should display first 10 lines of the file
r.take(10)

7- Launch PySpark
/usr/spark2.0.1/bin/pyspark

Running PySpark in Jupyter / IPython notebook | CloudxLab Blog

[Deepak_Sharma]

Thanks for the response. However using the instructions provided I was not able to bring up the Spark UI in the browser.Do I need to use the complete IP plus the port to bring the sparkUI ? Also I submitted a spark streaming job but I was not able to find any log or any thing related to Spark UI or Spark History Server.is there a way to find the link of these two (UI and History server).I tried to search in Resouce Manager UI there also no luck.

[abhinav]

Hi Deepak,

Let’s say your web console in on e.cloudxlab.com and your Spark job is running on port 4045.
To access Spark UI, you will have to type in http://e.cloudxlab.com:4045 in your browser

Also, Spark History Server for Spark 1.6 is located at http://c.cloudxlab.com:18080/ and for Spark 2.3 is located at http://c.cloudxlab.com:18081/

Hope this helps.

Thanks

[abhinav]

Please find port number in the console from where you are running Spark job

[Jeevankrishna_Rao]

the above steps to access Spark UI is not working, can you check this out with the latest steps…

[abhinav]

Hi @Jeevankrishna_Rao,

I have updated my answer to include new URLs and ports.

By the way, you can find the spark history server URLs in Ambari also.

Hope this helps.

Thanks

[vinod_paluvuri]

Hi @abhinav,
I tried what you have suggested. But unable to open spark web UI. Please help.

My spark shell started at 4045 port. and I am working on https://e.cloudxlab.com/. So i tried with http://e.cloudxlab.com:4045/...error message is unable to connect.

[vinpaluvuri5873@cxln4 ~]$ spark-shell
SPARK_MAJOR_VERSION is set to 2, using Spark2
Setting default log level to “WARN”.
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
19/08/02 03:02:07 WARN Utils: Service ‘SparkUI’ could not bind on port 4040. Attempting port 4041.
19/08/02 03:02:07 WARN Utils: Service ‘SparkUI’ could not bind on port 4041. Attempting port 4042.
19/08/02 03:02:07 WARN Utils: Service ‘SparkUI’ could not bind on port 4042. Attempting port 4043.
19/08/02 03:02:07 WARN Utils: Service ‘SparkUI’ could not bind on port 4043. Attempting port 4044.
19/08/02 03:02:07 WARN Utils: Service ‘SparkUI’ could not bind on port 4044. Attempting port 4045.
Spark context Web UI available at http://10.142.1.4:4045

Regards,
Vinod.

[sgiri]

Hi,
I am able to open the spark UI.

Please note that the IP address displayed is privated. From “My Lab” I found the corresponding public name which e.cloudxlab.com.

Then, I open the UI using the public name:

[spark]

But ip address is not in the mapped list (https://cloudxlab.com/my-lab#ip-mappings)
Here i attached both in one screenshots(due to limitation).
Can you let me know how to figure out spark web ui

[abhinav]

Hi,

Can you please check now? I have added the IP mapping

[spark]

I see new ip which is not in mapping. Is there anyway we figure out all domain names for corresponding ip address. Here i am attaching screenshot.

[spark]

I have see ip address(different one) but spark UI is not showing. Please find the attached screenshot.
Can you please let me know how to access

[pratikkawalgikar]

Hello All,

Thanks for the blog for accessing Web UI of Spark.
I have one question i.e. while we are creating rdd’s and performing transformations and actions on it through shell, the jobs can be viewed but why the jobs cannot be seen when we are doing it through Jupyter notebook?

[sgiri]

You just have to find out the port on which sparkUI is running. It is in the some environment variable that you can find from sc or spark entry point.

[Antariksh_K]

Hi @abhinav,

The UI is working when using spark-shell, but not working with spark-shell --master yarn. I tried all the ways mentioned in this thread.
Could you please help at the earliest.

Thanks

[Antariksh_K]

the UI link redirects to such link “http://cxln2.c.thelab-240901.internal:8088/proxy/redirect/application_1618218190443_4817/” and it doesn’t work

[Vagdevi_K]

I am able to access sparkUI. Just need to be careful about some petty things:

Am adding to Sandeep’s answer:

Please note that the IP address displayed is privated. From “My Lab” I found the corresponding public name which e.cloudxlab.com

Then, I open the UI using the public name:

NOTE:
(1) the URL should be http, not https
(2) in the link, the port number should be preceded by :, not a ‘/’. For example, it should be something like:

http://e.cloudxlab.com:4046/environment/

[Vijay_Hegde]

I am not able to access spark UI. I followed all the suggestions mention in this forum but none of them work.
http://e.cloudxlab.com:4044/

[sandeepgiri]

I just tried. It worked for me. Could you follow the instructions given by Vagdevi?