How do I access the WebUI from Jupyter?

#1

Hi,

I am using the Spark from Jupyter scala notebook (i.e. Apache Toree).

How to find out the URL of the spark UI?

Spark UI jobs through Jupyter notebook
#2

When you start the spark using either pyspark or spark-shell, it starts spark ui at a port. This port is by default 4040 but if it is busy it tries 4041. And then 4042 and so on. When you run spark-shell or pyspark from console it prints on the screen

But if you are using Spark from Jupyter which is also called Apache Toree, then it doesn’t print anything on the screen.

So, to access the Spark UI URl, you will have to run the following command in the Jupyter scala notebook: sc.uiWebUrl

To run something in notebook, type something in the box and press shift+enter.

Also, please note that, it prints the local host name. You would need to find the right hostname in CloudxLab’s “IP Mappings” tab.

Regards,
Sandeep Giri

1 Like