Hi,
below commanf was executed in webconsole :
hadoop jar /usr/hdp/2.3.4.0-3485/hadoop-mapreduce/hadoop-streaming.jar -D mapred.reduce.tasks=1 -input /user/mgurnani999463/mrinpu
t.txt -output /user/mgurnani999463/wc_cloudxlab_output.out -file /home/mgurnani999463/wordcount/wordcount_mapper.py -file /home/mgurnani999463/wordcount/wordcount_reducer.py -
mapper “python /home/mgurnani999463/wordcount/wordcount_mapper.py” -reducer “python /home/mgurnani999463/wordcount/wordcount_reducer.py”
std output:
WARNING: Use “yarn jar” to launch YARN applications.
18/02/10 13:47:30 WARN streaming.StreamJob: -file option is deprecated, please use generic option -files instead.
packageJobJar: [/home/mgurnani999463/wordcount/wordcount_mapper.py, /home/mgurnani999463/wordcount/wordcount_reducer.py] [/usr/hdp/2.3.4.0-3485/hadoop-mapreduce/hadoop-streami
ng-2.7.1.2.3.4.0-3485.jar] /tmp/streamjob3410259776003610953.jar tmpDir=null
18/02/10 13:47:31 INFO impl.TimelineClientImpl: Timeline service address: http://ip-172-31-13-154.ec2.internal:8188/ws/v1/timeline/
18/02/10 13:47:31 INFO client.RMProxy: Connecting to ResourceManager at ip-172-31-53-48.ec2.internal/172.31.53.48:8050
18/02/10 13:47:31 INFO impl.TimelineClientImpl: Timeline service address: http://ip-172-31-13-154.ec2.internal:8188/ws/v1/timeline/
18/02/10 13:47:31 INFO client.RMProxy: Connecting to ResourceManager at ip-172-31-53-48.ec2.internal/172.31.53.48:8050
18/02/10 13:47:32 INFO mapred.FileInputFormat: Total input paths to process : 1
18/02/10 13:47:32 INFO mapreduce.JobSubmitter: number of splits:2
18/02/10 13:47:32 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1517296050843_1654
18/02/10 13:47:32 INFO impl.YarnClientImpl: Submitted application application_1517296050843_1654
18/02/10 13:47:32 INFO mapreduce.Job: The url to track the job:
18/02/10 13:47:32 INFO mapreduce.Job: Running job: job_1517296050843_1654
18/02/10 13:47:39 INFO mapreduce.Job: Job job_1517296050843_1654 running in uber mode : false
18/02/10 13:47:39 INFO mapreduce.Job: map 0% reduce 0%
18/02/10 13:47:41 INFO mapreduce.Job: Task Id : attempt_1517296050843_1654_m_000001_1000, Status : FAILED
Could not find any valid local directory for nmPrivate/container_e123_1517296050843_1654_02_000002.tokens
18/02/10 13:47:43 INFO mapreduce.Job: Task Id : attempt_1517296050843_1654_m_000000_1000, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 2
at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:322)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:535)
at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)