Spark UI on CloudxLab

abhinav · September 11, 2018, 8:52am

I have updated my answer to include new URLs and ports.

By the way, you can find the spark history server URLs in Ambari also.

Hope this helps.

Thanks

vinod_paluvuri · August 5, 2019, 4:48am

Hi @abhinav,
I tried what you have suggested. But unable to open spark web UI. Please help.

My spark shell started at 4045 port. and I am working on https://e.cloudxlab.com/. So i tried with http://e.cloudxlab.com:4045/...error message is unable to connect.

[vinpaluvuri5873@cxln4 ~]$ spark-shell
SPARK_MAJOR_VERSION is set to 2, using Spark2
Setting default log level to “WARN”.
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
19/08/02 03:02:07 WARN Utils: Service ‘SparkUI’ could not bind on port 4040. Attempting port 4041.
19/08/02 03:02:07 WARN Utils: Service ‘SparkUI’ could not bind on port 4041. Attempting port 4042.
19/08/02 03:02:07 WARN Utils: Service ‘SparkUI’ could not bind on port 4042. Attempting port 4043.
19/08/02 03:02:07 WARN Utils: Service ‘SparkUI’ could not bind on port 4043. Attempting port 4044.
19/08/02 03:02:07 WARN Utils: Service ‘SparkUI’ could not bind on port 4044. Attempting port 4045.
Spark context Web UI available at http://10.142.1.4:4045

Regards,
Vinod.

sgiri · August 5, 2019, 5:28pm

Hi,
I am able to open the spark UI.

Please note that the IP address displayed is privated. From “My Lab” I found the corresponding public name which e.cloudxlab.com.

Then, I open the UI using the public name:

spark · April 18, 2020, 12:12pm

But ip address is not in the mapped list (https://cloudxlab.com/my-lab#ip-mappings)
Here i attached both in one screenshots(due to limitation).
Can you let me know how to figure out spark web ui

abhinav · April 20, 2020, 9:48am

Hi,

Can you please check now? I have added the IP mapping

spark · April 20, 2020, 6:58pm

I see new ip which is not in mapping. Is there anyway we figure out all domain names for corresponding ip address. Here i am attaching screenshot.

spark · April 21, 2020, 7:28am

I have see ip address(different one) but spark UI is not showing. Please find the attached screenshot.
Can you please let me know how to access

pratikkawalgikar · June 22, 2020, 12:01pm

Hello All,

Thanks for the blog for accessing Web UI of Spark.
I have one question i.e. while we are creating rdd’s and performing transformations and actions on it through shell, the jobs can be viewed but why the jobs cannot be seen when we are doing it through Jupyter notebook?

sgiri · June 28, 2020, 10:02pm

You just have to find out the port on which sparkUI is running. It is in the some environment variable that you can find from sc or spark entry point.

Antariksh_K · May 9, 2021, 6:51am

Hi @abhinav,

The UI is working when using spark-shell, but not working with spark-shell --master yarn. I tried all the ways mentioned in this thread.
Could you please help at the earliest.

Thanks

Antariksh_K · May 9, 2021, 7:02am

the UI link redirects to such link “http://cxln2.c.thelab-240901.internal:8088/proxy/redirect/application_1618218190443_4817/” and it doesn’t work

Vagdevi_K · May 28, 2021, 12:22pm

I am able to access sparkUI. Just need to be careful about some petty things:

Am adding to Sandeep’s answer:

Please note that the IP address displayed is privated. From “My Lab” I found the corresponding public name which e.cloudxlab.com

Then, I open the UI using the public name:

NOTE:
(1) the URL should be http, not https
(2) in the link, the port number should be preceded by :, not a ‘/’. For example, it should be something like:

http://e.cloudxlab.com:4046/environment/

Vijay_Hegde · May 8, 2022, 7:28am

I am not able to access spark UI. I followed all the suggestions mention in this forum but none of them work.
http://e.cloudxlab.com:4044/

sandeepgiri · May 9, 2022, 4:56am

I just tried. It worked for me. Could you follow the instructions given by Vagdevi?

sandeepgiri · May 9, 2022, 4:57am

Vijay_Hegde · May 9, 2022, 3:47pm

I am attaching the screen shot for your reference.

Vijay_Hegde · May 11, 2022, 9:40am

Team, can you be little swift in your replies. I have this problem for sometime now and stuck with my practical.

Vijay_Hegde · May 11, 2022, 9:41am

I am attaching the screen shot for your reference. Can you please look into this issue on a priority basis, I can not move ahead with my practical without having sparkUI.

Jayaprakashreddy_Alu · July 16, 2023, 3:01pm

I am unable to access spark Ui

Shubh_Tripathi · July 17, 2023, 11:32am

Hi,

You need to follow the following steps for it:

Open spark shell by running:

spark-shell

It will display some logs while starting the shell. In that there will be a log at fourth number like: ‘Spark context Web UI available at’. There will be mentioned a URL with the port number. You need to copy that port number and then check your lab URL.
Your lab URL will be either e.cloudxlab.com or f.cloudxlab.com. If it’s f, then you need to open:http://f.cloudxlab.com:port_number

If it’s e, then: http://e.cloudxlab.com:p ort_number.

The port_number will be one from your spark session.