DAG did not succeed due to VERTEX_FAILURE while insert overwrite table

Created a external table for partition.

While inserting data from another table (below is the query)

set hive.exec.dynamic.partition.mode=nonstrict;
from crime_et
insert overwrite table crime_et_pt
partition(cdate)
select cdate, address,district, beat, grid ,crimedescr ,ucr_ncic_code,latitude, longitude ;

Below is my error:

Status: Failed
Vertex failed, vertexName=Map 1, vertexId=vertex_1514380771546_0902_1_00, diagnostics=[Task failed, taskId=task_ 1514380771546_0902_1_00_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java. lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtim e Error while processing row {“cdate”:“1/11/2006 15:06”,“address”:“5310 79TH ST”,“district”:6,“beat”:"6C ",“grid”:1153,“crimedescr”:“TOWED/STORED VEHICLE”,“ucr_ncic_code”:7000,“latitude”:38.52745012,“longitude”:-121.4 120437}
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:181)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:172)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:172)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:168)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error whil e processing row {“cdate”:“1/11/2006 15:06”,“address”:“5310 79TH ST”,“district”:6,“beat”:"6C ",“grid”:115 3,“crimedescr”:“TOWED/STORED VEHICLE”,“ucr_ncic_code”:7000,“latitude”:38.52745012,“longitude”:-121.4120437}
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:91)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:326)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150)
… 14 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {“cdate”:“1 /11/2006 15:06”,“address”:“5310 79TH ST”,“district”:6,“beat”:"6C ",“grid”:1153,“crimedescr”:“TOWED/STORED VEHICLE”,“ucr_ncic_code”:7000,“latitude”:38.52745012,“longitude”:-121.4120437}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:545)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83)
… 17 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveFatalException: [Error 20004]: Fatal error occurred when node tried to create too many dynamic partitions. The maximum number of dynamic partitions is controlled by hive.exec .max.dynamic.partitions and hive.exec.max.dynamic.partitions.pernode. Maximum was set to: 2000
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.getDynOutPaths(FileSinkOperator.java:933)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:709)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:88)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:97)
at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:164)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:535)
… 18 more
], TaskAttempt 1 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeEx ception: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {“cdate”:“1/1 1/2006 15:06”,“address”:“5310 79TH ST”,“district”:6,“beat”:"6C ",“grid”:1153,“crimedescr”:“TOWED/STORED V EHICLE”,“ucr_ncic_code”:7000,“latitude”:38.52745012,“longitude”:-121.4120437}
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:181)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:172)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:172)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:168)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error whil e processing row {“cdate”:“1/11/2006 15:06”,“address”:“5310 79TH ST”,“district”:6,“beat”:"6C ",“grid”:115 3,“crimedescr”:“TOWED/STORED VEHICLE”,“ucr_ncic_code”:7000,“latitude”:38.52745012,“longitude”:-121.4120437}
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:91)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:326)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150)
… 14 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {“cdate”:“1 /11/2006 15:06”,“address”:“5310 79TH ST”,“district”:6,“beat”:"6C ",“grid”:1153,“crimedescr”:“TOWED/STORED VEHICLE”,“ucr_ncic_code”:7000,“latitude”:38.52745012,“longitude”:-121.4120437}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:545)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83)
… 17 more
]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1514380771546_090 2_1_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0

HI Sammsundar4905,

your getting the VERTEX_FAILURE due to the partition column having the date and time.

Two options you have for loading the data into the external table

Option 1:

When creating the external table have one more extra column as crime_date_time and try to use unix_timestamp

CREATE EXTERNAL TABLE crime_et_pt
** (**
address string,district string,beat string,grid string,crimedescr string,ucr_ncic_code string,latitude string,longitude string,cdate_time string
)
PARTITIONED BY (cdate string)
LOCATION ‘/user/user_name/crime_data_file_dir/’;

from crime_et
insert overwrite table crime_et_pt
partition(cdate)
select address,district,beat,grid,crimedescr,ucr_ncic_code,latitude,longitude,from_unixtime(unix_timestamp(cdate,‘MM/dd/yyyy HH:mm’),‘yyyy-MM-dd HH:mm’) as cdate_time,from_unixtime(unix_timestamp(cdate,‘MM/dd/yyyy HH:mm’),‘yyyy-MM-dd’) as cdate;

Option 2:

Not recommended :thinking:

Use mr as execution engine and increase the partitions which would take more time

set hive.execution.engine=mr;
set hive.exec.max.dynamic.partitions=8000;
set hive.exec.max.dynamic.partitions.pernode=8000;

Hopes it helps you :slight_smile:

Thanks
Shanmukh

1 Like

Thanks for your reply Shanmukh. But this solution applicable to this data only. Am getting the same error for all the data set(without date and time also ) when I try to create a partition.

Hi sammsundar,

Mostly you have to increase the partitions pernode

set hive.exec.max.dynamic.partitions.pernode=2000;

Let me know on what type of data columns your creating partitions.

You will have configuration settings before running query.

Thanks
Shanmukh

1 Like

I have increased the partition also and for another data set (has no date time) also. Still the same error exists.

Created an external table:

create external table winereview_et (sno int, country string, description string, designation string, points int, price int, province string, region_1 string, region_2 string, variety string, winery string)
row format delimited
fields terminated by ‘\t’
location ‘/user/sammsundar4905/externaltable/’;

Created a separate external table for partition:
CREATE external TABLE winereview_et_part (sno string, description string, designation string, points string, price string, province string, region_1 string, region_2 string, variety string, winery string)
COMMENT ‘A partitened copy of winereview_et’
PARTITIONED BY(country string)
location ‘/user/sammsundar4905/externaltable/’;

Inserting data to external partition table:
FROM winereview_et
INSERT OVERWRITE TABLE winereview_et_part
PARTITION (country)
SELECT sno,country,description,designation,points,price,province,region_1,region_2,variety,winery;

While inserting, facing the below error:
Total jobs = 1
Launching Job 1 out of 1

Status: Running (Executing on YARN cluster with App id application_1514380771546_1401)


    VERTICES      STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED

Map 1 FAILED 3 0 0 3 10 0

VERTICES: 00/01 [>>--------------------------] 0% ELAPSED TIME: 267.44 s

Status: Failed
Vertex failed, vertexName=Map 1, vertexId=vertex_1514380771546_1401_2_00, diagnostics=[Task failed, taskId=task_1514380771546_1401_2_00_000002, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {“sno”:106250,“country”:“US”,“description”:"“Simple and oaky, with woody, caramelized flavors masking a thin wine. Dry and acidic, its watery flavors barely suggest tangerines.”",“designation”:"",“points”:82,“price”:15,“province”:“California”,“region_1”:“Calaveras County”,“region_2”:“Sierra Foothills”,“variety”:“Marsanne-Roussanne”,“winery”:“Albeno Munari”}
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:181)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:172)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:172)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:168)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {“sno”:106250,“country”:“US”,“description”:"“Simple and oaky, with woody, caramelized flavors masking a thin wine. Dry and acidic, its watery flavors barely suggest tangerines.”",“designation”:"",“points”:82,“price”:15,“province”:“California”,“region_1”:“Calaveras County”,“region_2”:“Sierra Foothills”,“variety”:“Marsanne-Roussanne”,“winery”:“Albeno Munari”}
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:91)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:326)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150)
… 14 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {“sno”:106250,“country”:“US”,“description”:"“Simple and oaky, with woody, caramelized flavors masking a thin wine. Dry and acidic, its watery flavors barely suggest tangerines.”",“designation”:"",“points”:82,“price”:15,“province”:“California”,“region_1”:“Calaveras County”,“region_2”:“Sierra Foothills”,“variety”:“Marsanne-Roussanne”,“winery”:“Albeno Munari”}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:545)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83)
… 17 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveFatalException: [Error 20004]: Fatal error occurred when node tried to create too many dynamic partitions. The maximum number of dynamic partitions is controlled by hive.exec.max.dynamic.partitions and hive.exec.max.dynamic.partitions.pernode. Maximum was set to: 2000
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.getDynOutPaths(FileSinkOperator.java:933)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:709)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:88)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:97)
at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:164)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:535)
… 18 more
], TaskAttempt 1 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {“sno”:106250,“country”:“US”,“description”:"“Simple and oaky, with woody, caramelized flavors masking a thin wine. Dry and acidic, its watery flavors barely suggest tangerines.”",“designation”:"",“points”:82,“price”:15,“province”:“California”,“region_1”:“Calaveras County”,“region_2”:“Sierra Foothills”,“variety”:“Marsanne-Roussanne”,“winery”:“Albeno Munari”}
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:181)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:172)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:172)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:168)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {“sno”:106250,“country”:“US”,“description”:"“Simple and oaky, with woody, caramelized flavors masking a thin wine. Dry and acidic, its watery flavors barely suggest tangerines.”",“designation”:"",“points”:82,“price”:15,“province”:“California”,“region_1”:“Calaveras County”,“region_2”:“Sierra Foothills”,“variety”:“Marsanne-Roussanne”,“winery”:“Albeno Munari”}
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:91)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:326)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150)
… 14 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {“sno”:106250,“country”:“US”,“description”:"“Simple and oaky, with woody, caramelized flavors masking a thin wine. Dry and acidic, its watery flavors barely suggest tangerines.”",“designation”:"",“points”:82,“price”:15,“province”:“California”,“region_1”:“Calaveras County”,“region_2”:“Sierra Foothills”,“variety”:“Marsanne-Roussanne”,“winery”:“Albeno Munari”}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:545)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83)
… 17 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveFatalException: [Error 20004]: Fatal error occurred when node tried to create too many dynamic partitions. The maximum number of dynamic partitions is controlled by hive.exec.max.dynamic.partitions and hive.exec.max.dynamic.partitions.pernode. Maximum was set to: 2000
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.getDynOutPaths(FileSinkOperator.java:933)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:709)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:88)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:97)
at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:164)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:535)
… 18 more
], TaskAttempt 2 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {“sno”:106250,“country”:“US”,“description”:"“Simple and oaky, with woody, caramelized flavors masking a thin wine. Dry and acidic, its watery flavors barely suggest tangerines.”",“designation”:"",“points”:82,“price”:15,“province”:“California”,“region_1”:“Calaveras County”,“region_2”:“Sierra Foothills”,“variety”:“Marsanne-Roussanne”,“winery”:“Albeno Munari”}
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:181)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:172)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:172)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:168)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {“sno”:106250,“country”:“US”,“description”:"“Simple and oaky, with woody, caramelized flavors masking a thin wine. Dry and acidic, its watery flavors barely suggest tangerines.”",“designation”:"",“points”:82,“price”:15,“province”:“California”,“region_1”:“Calaveras County”,“region_2”:“Sierra Foothills”,“variety”:“Marsanne-Roussanne”,“winery”:“Albeno Munari”}
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:91)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:326)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150)
… 14 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {“sno”:106250,“country”:“US”,“description”:"“Simple and oaky, with woody, caramelized flavors masking a thin wine. Dry and acidic, its watery flavors barely suggest tangerines.”",“designation”:"",“points”:82,“price”:15,“province”:“California”,“region_1”:“Calaveras County”,“region_2”:“Sierra Foothills”,“variety”:“Marsanne-Roussanne”,“winery”:“Albeno Munari”}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:545)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83)
… 17 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveFatalException: [Error 20004]: Fatal error occurred when node tried to create too many dynamic partitions. The maximum number of dynamic partitions is controlled by hive.exec.max.dynamic.partitions and hive.exec.max.dynamic.partitions.pernode. Maximum was set to: 2000
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.getDynOutPaths(FileSinkOperator.java:933)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:709)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:88)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:97)
at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:164)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:535)
… 18 more
], TaskAttempt 3 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {“sno”:106250,“country”:“US”,“description”:"“Simple and oaky, with woody, caramelized flavors masking a thin wine. Dry and acidic, its watery flavors barely suggest tangerines.”",“designation”:"",“points”:82,“price”:15,“province”:“California”,“region_1”:“Calaveras County”,“region_2”:“Sierra Foothills”,“variety”:“Marsanne-Roussanne”,“winery”:“Albeno Munari”}
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:181)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:172)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:172)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:168)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {“sno”:106250,“country”:“US”,“description”:"“Simple and oaky, with woody, caramelized flavors masking a thin wine. Dry and acidic, its watery flavors barely suggest tangerines.”",“designation”:"",“points”:82,“price”:15,“province”:“California”,“region_1”:“Calaveras County”,“region_2”:“Sierra Foothills”,“variety”:“Marsanne-Roussanne”,“winery”:“Albeno Munari”}
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:91)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:326)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150)
… 14 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {“sno”:106250,“country”:“US”,“description”:"“Simple and oaky, with woody, caramelized flavors masking a thin wine. Dry and acidic, its watery flavors barely suggest tangerines.”",“designation”:"",“points”:82,“price”:15,“province”:“California”,“region_1”:“Calaveras County”,“region_2”:“Sierra Foothills”,“variety”:“Marsanne-Roussanne”,“winery”:“Albeno Munari”}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:545)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83)
… 17 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveFatalException: [Error 20004]: Fatal error occurred when node tried to create too many dynamic partitions. The maximum number of dynamic partitions is controlled by hive.exec.max.dynamic.partitions and hive.exec.max.dynamic.partitions.pernode. Maximum was set to: 2000
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.getDynOutPaths(FileSinkOperator.java:933)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:709)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:88)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:97)
at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:164)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:535)
… 18 more
]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:2, Vertex vertex_1514380771546_1401_2_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE]
DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1514380771546_1401_2_00, diagnostics=[Task failed, taskId=task_1514380771546_1401_2_00_000002, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {“sno”:106250,“country”:“US”,“description”:"“Simple and oaky, with woody, caramelized flavors masking a thin wine. Dry and acidic, its watery flavors barely suggest tangerines.”",“designation”:"",“points”:82,“price”:15,“province”:“California”,“region_1”:“Calaveras County”,“region_2”:“Sierra Foothills”,“variety”:“Marsanne-Roussanne”,“winery”:“Albeno Munari”}
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:181)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:172)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:172)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:168)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {“sno”:106250,“country”:“US”,“description”:"“Simple and oaky, with woody, caramelized flavors masking a thin wine. Dry and acidic, its watery flavors barely suggest tangerines.”",“designation”:"",“points”:82,“price”:15,“province”:“California”,“region_1”:“Calaveras County”,“region_2”:“Sierra Foothills”,“variety”:“Marsanne-Roussanne”,“winery”:“Albeno Munari”}
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:91)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:326)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150)
… 14 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {“sno”:106250,“country”:“US”,“description”:"“Simple and oaky, with woody, caramelized flavors masking a thin wine. Dry and acidic, its watery flavors barely suggest tangerines.”",“designation”:"",“points”:82,“price”:15,“province”:“California”,“region_1”:“Calaveras County”,“region_2”:“Sierra Foothills”,“variety”:“Marsanne-Roussanne”,“winery”:“Albeno Munari”}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:545)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83)
… 17 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveFatalException: [Error 20004]: Fatal error occurred when node tried to create too many dynamic partitions. The maximum number of dynamic partitions is controlled by hive.exec.max.dynamic.partitions and hive.exec.max.dynamic.partitions.pernode. Maximum was set to: 2000
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.getDynOutPaths(FileSinkOperator.java:933)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:709)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:88)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:97)
at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:164)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:535)
… 18 more
], TaskAttempt 1 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {“sno”:106250,“country”:“US”,“description”:"“Simple and oaky, with woody, caramelized flavors masking a thin wine. Dry and acidic, its watery flavors barely suggest tangerines.”",“designation”:"",“points”:82,“price”:15,“province”:“California”,“region_1”:“Calaveras County”,“region_2”:“Sierra Foothills”,“variety”:“Marsanne-Roussanne”,“winery”:“Albeno Munari”}
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:181)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:172)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:172)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:168)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {“sno”:106250,“country”:“US”,“description”:"“Simple and oaky, with woody, caramelized flavors masking a thin wine. Dry and acidic, its watery flavors barely suggest tangerines.”",“designation”:"",“points”:82,“price”:15,“province”:“California”,“region_1”:“Calaveras County”,“region_2”:“Sierra Foothills”,“variety”:“Marsanne-Roussanne”,“winery”:“Albeno Munari”}
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:91)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:326)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150)
… 14 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {“sno”:106250,“country”:“US”,“description”:"“Simple and oaky, with woody, caramelized flavors masking a thin wine. Dry and acidic, its watery flavors barely suggest tangerines.”",“designation”:"",“points”:82,“price”:15,“province”:“California”,“region_1”:“Calaveras County”,“region_2”:“Sierra Foothills”,“variety”:“Marsanne-Roussanne”,“winery”:“Albeno Munari”}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:545)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83)
… 17 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveFatalException: [Error 20004]: Fatal error occurred when node tried to create too many dynamic partitions. The maximum number of dynamic partitions is controlled by hive.exec.max.dynamic.partitions and hive.exec.max.dynamic.partitions.pernode. Maximum was set to: 2000
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.getDynOutPaths(FileSinkOperator.java:933)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:709)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:88)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:97)
at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:164)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:535)
… 18 more

]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:2, Vertex vertex_1514380771546_1401_2_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0

1 Like

Hi @sammsundar4905

The total no. of countries in the world < 200 which is less than the default partitions

The input data which your trying to put into the partition table must be incorrect

Check your input table(winereview_et) on all columns you have data or anyone column has entire data which may cause the issue.

You need to check the partition column and do a data cleansing to have the correct partitions in place and also selecting the correct partition column in table helps you not to have issues while inserting the data into the partition tables.

if you still face the issue i can help you. :slight_smile:

Thanks,
Shanmukh

1 Like

Thank you so much for the excellent solution. helped me a lot and easy to understand for me. keep it up.

Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.IllegalArgumentException: field ended by ‘;’: expected ‘;’ but got ‘note’ at line 45: optional binary internal note
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:829)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:1004)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:937)
at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:937)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:128)
at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:152)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:552)
… 20 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.IllegalArgumentException: field ended by ‘;’: expected ‘;’ but got ‘note’ at line 45: optional binary internal note
at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:282)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketForFileIdx(FileSinkOperator.java:872)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:823)
… 27 more
Caused by: java.lang.IllegalArgumentException: field ended by ‘;’: expected ‘;’ but got ‘note’ at line 45: optional binary internal note
at org.apache.parquet.schema.MessageTypeParser.check(MessageTypeParser.java:239)
at org.apache.parquet.schema.MessageTypeParser.addPrimitiveType(MessageTypeParser.java:208)
at org.apache.parquet.schema.MessageTypeParser.addType(MessageTypeParser.java:113)
at org.apache.parquet.schema.MessageTypeParser.addGroupTypeFields(MessageTypeParser.java:101)
at org.apache.parquet.schema.MessageTypeParser.parse(MessageTypeParser.java:94)
at org.apache.parquet.schema.MessageTypeParser.parseMessageType(MessageTypeParser.java:84)
at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.getSchema(DataWritableWriteSupport.java:52)
at org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.init(DataWritableWriteSupport.java:58)
at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:418)
at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:380)
at org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.(ParquetRecordWriterWrapper.java:70)
at org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat.getParquerRecordWriterWrapper(MapredParquetOutputFormat.java:137)
at org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat.getHiveRecordWriter(MapredParquetOutputFormat.java:126)
at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordWriter(HiveFileFormatUtils.java:294)
at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:279)
… 29 more
]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:1, Vertex vertex_1675928979268_0187_1_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE]Vertex killed, vertexName=Reducer 2, vertexId=vertex_1675928979268_0187_1_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:1, Vertex vertex_1675928979268_0187_1_01 [Reducer 2] killed/failed due to:OTHER_VERTEX_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1 (state=08S01,code=2)

Hi,

Which command did you run?