Hi,
Will the spark applications which are executed on local be shown in the history server?
How to find the link for the history server ?
I want to look at a simple spark command executed locally using spark-shell
On the logs it is mentioned that
Spark context Web UI available at http://172.31.20.58:4042 But when I go to it it says
“website not available”
Is “spark web ui” == “spark ui” ?
Thanks.
Hi Utkarsh,
When you launch spark-shell with yarn then it will reflect in the YARN’s UI.
You can try the following command: spark-shell --master yarn
This would show up Hue’s job browser.
Spark context Web UI available at http://172.31.20.58:4042 But when I go to it it says
“website not available”
Since the IP Address “172.31.20.58” is private, you will not be able to access from outside. You can find the corresponding public domain name using the “My Lab”.
Then, you can replace the IP adress with public url. In your case it is g.cloudxlab.com. So, the URL is would have been http://g.cloudxlab.com:4042
Are the locally executed spark jobs not shown on the UI ?
Also is the UI available only when the job is being executed ?
Yes, the locally executed jobs (i.e. without --master yarn ) don’t talk to hadoop and therefore the UI is not hooked into HUE.
When you run spark-shell in local mode (without --master yarn) it starts up it Spark UI and try to first open 4040 then 4041 … and so on. You can access that UI as mentioned above.
