Spark UI jobs through Jupyter notebook

Technical Discussions Apache Spark
#1

Hello All,

I have one question, while we are creating rdd’s and performing transformations and actions on it through shell, the jobs can be viewed in Spark UI but why the jobs cannot be seen when we are doing it through Jupyter notebook?

#2

Hi,

We can go to SparkUI for seeing the jobs which are running via Jupyter. Spark UI is default interface for checking status job, wether it is run from shell or from Jupyter

#3

Hi Abhinav,

Thanks for your reply.
I created an RDD and applied the some transformations and actions through Jupyter notebook, but I’m unable to see the same in Spark UI. Can you please let me know how to proceed?

#4

Can you please check on which port your job is running in notebook. I do not remember the command for the same but in notebook you can find the port on which your job is running.

After you get the port, please go to that port on your web browser and you can see the job status.

Hope this helps.

#5

Hi Abhinav,

Thanks for your support.
I found this post from cloudxlab forum: How do I access the WebUI from Jupyter?
Now I’m able to view Jupyter notebook jobs.

1 Like