问题描述：
hive 查询hudi 报错，
已将hudi 对应包hudi-hadoop-mr-bundle-0.9.0.jar放入lib目录下并重启hive
select count(1) from table ;
ERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1642524380391_3302_1_00, diagnostics=[Vertex vertex_1642524380391_3302_1_00 [Map 1] killed/failed due to:INIT_FAILURE, Fail to create InputInitializerManager, org.apache.tez.dag.api.TezReflectionException: Unable to instantiate class with 1 arguments: org.apache.hadoop.hive.ql.exec.tez.HiveSplitGeneratorat org.mon.ReflectionUtils.getNewInstance(ReflectionUtils.java:71)at org.mon.ReflectionUtils.createClazzInstance(ReflectionUtils.java:89)at org.apache.tez.dag.app.dag.RootInputInitializerManager$1.run(RootInputInitializerManager.java:152)at org.apache.tez.dag.app.dag.RootInputInitializerManager$1.run(RootInputInitializerManager.java:148)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:422)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)at org.apache.tez.dag.app.dag.RootInputInitializerManager.createInitializer(RootInputInitializerManager.java:148)at org.apache.tez.dag.app.dag.RootInputInitializerManager.runInputInitializers(RootInputInitializerManager.java:121)at org.apache.tez.dag.app.dag.impl.VertexImpl.setupInputInitializerManager(VertexImpl.java:4123)at org.apache.tez.dag.app.dag.impl.VertexImpl.access$3100(VertexImpl.java:208)at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.handleInitEvent(VertexImpl.java:2933)at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.transition(VertexImpl.java:2880)at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.transition(VertexImpl.java:2862)at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)at org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)at org.apache.tez.state.StateMachineTez.doTransition(StateMachineTez.java:59)at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:1958)at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:207)at org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:2317)at org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:2303)at org.mon.AsyncDispatcher.dispatch(AsyncDispatcher.java:180)at org.mon.AsyncDispatcher$1.run(AsyncDispatcher.java:115)at java.lang.Thread.run(Thread.java:748)Caused by: java.lang.reflect.InvocationTargetExceptionat sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newInstance(Constructor.java:423)at org.mon.ReflectionUtils.getNewInstance(ReflectionUtils.java:68)... 25 moreCaused by: java.lang.RuntimeException: Failed to load plan: hdfs://xxx/tmp/hive/root/8344ec71-67c8-4733-8dd1-41123f1e1729/hive_-02-08_16-54-15_006_1554035332667755211-922/root/_tez_scratch_dir/fcdb3e29-0d57-40cc-8bfe-21cfa9490cd6/map.xmlat org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:509)at org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:342)at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.<init>(HiveSplitGenerator.java:137)... 30 moreCaused by: org..esotericsoftware.kryo.KryoException: Unable to find class: org.apache.hudi.hadoop.HoodieParquetInputFormatSerialization trace:inputFileFormatClass (org.apache.hadoop.hive.ql.plan.PartitionDesc)aliasToPartnInfo (org.apache.hadoop.hive.ql.plan.MapWork)at org..esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:156)at org..esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:133)at org..esotericsoftware.kryo.Kryo.readClass(Kryo.java:670)at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClass(SerializationUtilities.java:185)at org..esotericsoftware.kryo.serializers.DefaultSerializers$ClassSerializer.read(DefaultSerializers.java:326)at org..esotericsoftware.kryo.serializers.DefaultSerializers$ClassSerializer.read(DefaultSerializers.java:314)at org..esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:759)at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObjectOrNull(SerializationUtilities.java:203)at org..esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:132)at org..esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)at org..esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:180)at org..esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:161)at org..esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:39)at org..esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:218)at org..esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)at org..esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)at org..esotericsoftware.kryo.Kryo.readObject(Kryo.java:686)at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:210)at org.apache.hadoop.hive.ql.exec.SerializationUtilities.deserializeObjectByKryo(SerializationUtilities.java:707)at org.apache.hadoop.hive.ql.exec.SerializationUtilities.deserializePlan(SerializationUtilities.java:613)at org.apache.hadoop.hive.ql.exec.SerializationUtilities.deserializePlan(SerializationUtilities.java:590)at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:470)... 32 moreCaused by: java.lang.ClassNotFoundException: org.apache.hudi.hadoop.HoodieParquetInputFormatat .URLClassLoader.findClass(URLClassLoader.java:382)at java.lang.ClassLoader.loadClass(ClassLoader.java:418)at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)at java.lang.ClassLoader.loadClass(ClassLoader.java:351)at java.lang.Class.forName0(Native Method)at java.lang.Class.forName(Class.java:348)at org..esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:154)... 55 more]
解决办法:
hive lib目录同级目录下创建auxlib，并将包放入此目录后 重启Hive
必须要auxlib目录
select count(1) from document_ro; INFO : Compiling command(queryId=hive_2029132712_713df3f8-91ab-46a2-8e1a-17bda938a644): select count(1) from document_roINFO : Semantic Analysis Completed (retrial = false)INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0, type:bigint, comment:null)], properties:null)INFO : Completed compiling command(queryId=hive_2029132712_713df3f8-91ab-46a2-8e1a-17bda938a644); Time taken: 1.619 secondsINFO : Executing command(queryId=hive_2029132712_713df3f8-91ab-46a2-8e1a-17bda938a644): select count(1) from document_roINFO : Query ID = hive_2029132712_713df3f8-91ab-46a2-8e1a-17bda938a644INFO : Total jobs = 1INFO : Launching Job 1 out of 1INFO : Starting task [Stage-1:MAPRED] in serial modeINFO : Subscribed to counters: [] for queryId: hive_2029132712_713df3f8-91ab-46a2-8e1a-17bda938a644INFO : Tez session hasn't been created yet. Opening sessionINFO : Dag name: select count(1) from document_ro (Stage-1)INFO : Status: Running (Executing on YARN cluster with App id application_1644382793459_0002)----------------------------------------------------------------------------------------------VERTICESMODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED ----------------------------------------------------------------------------------------------Map 1 .......... containerSUCCEEDED33 0 0 0 0 Reducer 2 ...... containerSUCCEEDED11 0 0 0 0 ----------------------------------------------------------------------------------------------VERTICES: 02/02 [==========================>>] 100% ELAPSED TIME: 3.55 s----------------------------------------------------------------------------------------------INFO : Status: DAG finished successfully in 3.45 secondsINFO : INFO : Query Execution SummaryINFO : ----------------------------------------------------------------------------------------------INFO : OPERATION DURATIONINFO : ----------------------------------------------------------------------------------------------INFO : Compile Query 1.62sINFO : Prepare Plan 4.15sINFO : Get Query Coordinator (AM) 0.01sINFO : Submit Plan 0.23sINFO : Start DAG 0.89sINFO : Run DAG 3.45sINFO : ----------------------------------------------------------------------------------------------INFO : INFO : Task Execution SummaryINFO : ----------------------------------------------------------------------------------------------INFO : VERTICESDURATION(ms) CPU_TIME(ms) GC_TIME(ms) INPUT_RECORDS OUTPUT_RECORDSINFO : ----------------------------------------------------------------------------------------------INFO :Map 1 1925.00 17,270 12631,6233INFO : Reducer 2 1.00 600 030INFO : ----------------------------------------------------------------------------------------------INFO : INFO : org.mon.counters.DAGCounter:INFO : NUM_SUCCEEDED_TASKS: 4INFO : TOTAL_LAUNCHED_TASKS: 4INFO : DATA_LOCAL_TASKS: 1INFO : RACK_LOCAL_TASKS: 2INFO : AM_CPU_MILLISECONDS: 3110INFO : AM_GC_TIME_MILLIS: 0INFO : File System Counters:INFO : FILE_BYTES_READ: 57INFO : FILE_BYTES_WRITTEN: 174INFO : HDFS_BYTES_READ: 48409293INFO : HDFS_BYTES_WRITTEN: 105INFO : HDFS_READ_OPS: 303INFO : HDFS_WRITE_OPS: 2INFO : HDFS_OP_CREATE: 1INFO : HDFS_OP_GET_FILE_STATUS: 153INFO : HDFS_OP_OPEN: 150INFO : HDFS_OP_RENAME: 1INFO : org.mon.counters.TaskCounter:INFO : SPILLED_RECORDS: 0INFO : NUM_SHUFFLED_INPUTS: 3INFO : NUM_FAILED_SHUFFLE_INPUTS: 0INFO : GC_TIME_MILLIS: 126INFO : TASK_DURATION_MILLIS: 5607INFO : CPU_MILLISECONDS: 17870INFO : PHYSICAL_MEMORY_BYTES: 4219469824INFO : VIRTUAL_MEMORY_BYTES: 36327419904INFO : COMMITTED_HEAP_BYTES: 4219469824INFO : INPUT_RECORDS_PROCESSED: 82INFO : INPUT_SPLIT_LENGTH_BYTES: 26237450INFO : OUTPUT_RECORDS: 3INFO : OUTPUT_LARGE_RECORDS: 0INFO : OUTPUT_BYTES: 12INFO : OUTPUT_BYTES_WITH_OVERHEAD: 36INFO : OUTPUT_BYTES_PHYSICAL: 150INFO : ADDITIONAL_SPILLS_BYTES_WRITTEN: 0INFO : ADDITIONAL_SPILLS_BYTES_READ: 0INFO : ADDITIONAL_SPILL_COUNT: 0INFO : SHUFFLE_BYTES: 78INFO : SHUFFLE_BYTES_DECOMPRESSED: 36INFO : SHUFFLE_BYTES_TO_MEM: 53INFO : SHUFFLE_BYTES_TO_DISK: 0INFO : SHUFFLE_BYTES_DISK_DIRECT: 25INFO : SHUFFLE_PHASE_TIME: 193INFO : FIRST_EVENT_RECEIVED: 91INFO : LAST_EVENT_RECEIVED: 190INFO : HIVE:INFO : CREATED_FILES: 1INFO : DESERIALIZE_ERRORS: 0INFO : RECORDS_IN_Map_1: 31623INFO : RECORDS_OUT_0: 1INFO : RECORDS_OUT_INTERMEDIATE_Map_1: 3INFO : RECORDS_OUT_INTERMEDIATE_Reducer_2: 0INFO : RECORDS_OUT_OPERATOR_FS_11: 1INFO : RECORDS_OUT_OPERATOR_GBY_10: 1INFO : RECORDS_OUT_OPERATOR_GBY_8: 3INFO : RECORDS_OUT_OPERATOR_MAP_0: 0INFO : RECORDS_OUT_OPERATOR_RS_9: 3INFO : RECORDS_OUT_OPERATOR_SEL_7: 31623INFO : RECORDS_OUT_OPERATOR_TS_0: 31623INFO : TaskCounter_Map_1_INPUT_document_ro:INFO : INPUT_RECORDS_PROCESSED: 79INFO : INPUT_SPLIT_LENGTH_BYTES: 26237450INFO : TaskCounter_Map_1_OUTPUT_Reducer_2:INFO : ADDITIONAL_SPILLS_BYTES_READ: 0INFO : ADDITIONAL_SPILLS_BYTES_WRITTEN: 0INFO : ADDITIONAL_SPILL_COUNT: 0INFO : OUTPUT_BYTES: 12INFO : OUTPUT_BYTES_PHYSICAL: 150INFO : OUTPUT_BYTES_WITH_OVERHEAD: 36INFO : OUTPUT_LARGE_RECORDS: 0INFO : OUTPUT_RECORDS: 3INFO : SPILLED_RECORDS: 0INFO : TaskCounter_Reducer_2_INPUT_Map_1:INFO : FIRST_EVENT_RECEIVED: 91INFO : INPUT_RECORDS_PROCESSED: 3INFO : LAST_EVENT_RECEIVED: 190INFO : NUM_FAILED_SHUFFLE_INPUTS: 0INFO : NUM_SHUFFLED_INPUTS: 3INFO : SHUFFLE_BYTES: 78INFO : SHUFFLE_BYTES_DECOMPRESSED: 36INFO : SHUFFLE_BYTES_DISK_DIRECT: 25INFO : SHUFFLE_BYTES_TO_DISK: 0INFO : SHUFFLE_BYTES_TO_MEM: 53INFO : SHUFFLE_PHASE_TIME: 193INFO : TaskCounter_Reducer_2_OUTPUT_out_Reducer_2:INFO : OUTPUT_RECORDS: 0INFO : org.apache.hadoop.hive.ql.exec.tez.HiveInputCounters:INFO : GROUPED_INPUT_SPLITS_Map_1: 3INFO : INPUT_DIRECTORIES_Map_1: 11INFO : INPUT_FILES_Map_1: 50INFO : RAW_INPUT_SPLITS_Map_1: 50INFO : Completed executing command(queryId=hive_2029132712_713df3f8-91ab-46a2-8e1a-17bda938a644); Time taken: 8.802 secondsINFO : OK+--------+| _c0 |+--------+| 31623 |+--------+1 row selected (10.553 seconds)
如果觉得《hive查询hudi表使用count报java.lang.ClassNotFoundException: org.apache.hudi.hadoop.HoodieParquetInputFormat》对你有帮助，请点赞、收藏，并留下你的观点哦！
