Yarn Resource Manager down

devkish · October 7, 2019, 10:53am

YARN resource manger is down. Request you to please make it up

sgiri · October 7, 2019, 11:07am

Thank you for letting us know. I have turned it on.

Please carry on your work.

In the meantime, I am debugging why has it gone down.

devkish · October 7, 2019, 3:46pm

YARN resource manger is again down. Request you to please make it up

sgiri · October 7, 2019, 4:07pm

Ouch! I have restarted it.

devkish · October 7, 2019, 4:08pm

Ok. Thank you for timely help

devkish · October 8, 2019, 2:11pm

YARN is down now. Can you please make it up?

S.R_Shinde · October 8, 2019, 2:24pm

same here…yarn is down!!

Mayur_Nathb · October 8, 2019, 2:26pm

Unable to work with hive or pig. Please find the below error

2019-10-08 14:17:42,094 [JobControl] INFO org.apache.hadoop.yarn.client.AHSProxy - Connecting to Application History server at cxln2.c.thelab-240901.internal/10.142.1.2:10200
2019-10-08 14:17:43,146 [JobControl] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: cxln2.c.thelab-240901.internal/10.142.1.2:8050. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2019-10-08 14:17:44,148 [JobControl] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: cxln2.c.thelab-240901.internal/10.142.1.2:8050. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2019-10-08 14:17:45,149 [JobControl] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: cxln2.c.thelab-240901.internal/10.142.1.2:8050. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2019-10-08 14:17:46,150 [JobControl] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: cxln2.c.thelab-240901.internal/10.142.1.2:8050. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2019-10-08 14:17:47,151 [JobControl] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: cxln2.c.thelab-240901.internal/10.142.1.2:8050. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2019-10-08 14:17:48,152 [JobControl] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: cxln2.c.thelab-240901.internal/10.142.1.2:8050. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2019-10-08 14:17:49,153 [JobControl] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: cxln2.c.thelab-240901.internal/10.142.1.2:8050. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2019-10-08 14:17:50,154 [JobControl] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: cxln2.c.thelab-240901.internal/10.142.1.2:8050. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2019-10-08 14:17:51,155 [JobControl] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: cxln2.c.thelab-240901.internal/10.142.1.2:8050. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2019-10-08 14:17:52,156 [JobControl] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: cxln2.c.thelab-240901.internal/10.142.1.2:8050. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
^C2019-10-08 14:17:53,157 [JobControl] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: cxln2.c.thelab-240901.internal/10.142.1.2:8050. Already tried 10 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2019-10-08 14:17:54,159 [JobControl] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: cxln2.c.thelab-240901.internal/10.142.1.2:8050. Already tried 11 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2019-10-08 14:17:55,160 [JobControl] INFO org.apache.hadoop.ipc.Client - Retrying connect to server: cxln2.c.thelab-240901.internal/10.142.1.2:8050. Already tried 12 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)

Gowtham_Babu · October 8, 2019, 2:59pm

How to check Yarn Resource Manager is Up or down? I am in still learning phase

devkish · October 8, 2019, 3:01pm

Go to http://a.cloudxlab.com:8080/#/main/services/YARN/summary . You can see that resource manager in red (stopped)

Gowtham_Babu · October 8, 2019, 5:02pm

Resource Manager is down.
Kindly make it UP.

Thanks & Regards,
Gowtham.

Gowtham_Babu · October 8, 2019, 5:03pm

Thanks for quick response.

Priya_Mahalankar · October 8, 2019, 6:36pm

This has been happening very frequently, Disappointing!!!

Please fix this ASAP,

Santhoshkumar_Thirum · October 8, 2019, 6:38pm

@sgiri - Hi Sandeep the YARN is down. Can you please restart?

Priya_Mahalankar · October 8, 2019, 6:50pm

is there any fixed support time ? and not 24*7 ?

sgiri · October 8, 2019, 7:23pm

Started resource manager.

Priya_Mahalankar · October 8, 2019, 7:54pm

@sgiri YARN is again down

S.R_Shinde · October 9, 2019, 7:31am

YARN is again down…please help us in this regard…!!!

Akhilesh_Koul · October 9, 2019, 8:14am

sgiri · October 9, 2019, 12:06pm

After upscaling the machine, it took a while to bring up the services.