近期准备上线了一个基于HBase的hive,但是通过spark查询这个hive表的时候出现了如下异常:
19/12/21 15:27:18 ERROR log: error in initSerDe: java.lang.ClassNotFoundException Class org.apache.hadoop.hive.hbase.HBaseSerDe not found
java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.hbase.HBaseSerDe not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2409)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:442)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:435)
at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:280)
at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:262)
at org.apache.hadoop.hive.ql.metadata.Table.getColsInternal(Table.java:632)
at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:615)
at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$getTableOption$1$$anonfun$apply$7.apply(HiveClientImpl.scala:379)
出错的原因是没有找到HBaseSerDe,这个类是在hive-hbase-handler的jar包中,具体目录对应CDH版本的位置在/opt/cloudera/parcels/CDH/jars目录下面
[root@data2 jars]# pwd
/opt/cloudera/parcels/CDH/jars
[root@data2 jars]# ls -l|grep hive-hbase
-rw-r--r-- 1 root root 119117 Jul 19 01:55 hive-hbase-handler-2.1.1-cdh6.3.0.jar
在spark目录下建立一个ln指向这个jar包,问题解决
[root@data3 ~]# cd /opt/cloudera/parcels/CDH/lib/spark/jars
[root@data3 jars]# ln -s /opt/cloudera/parcels/CDH/jars/hive-hbase-handler-2.1.1-cdh6.3.0.jar hive-hbase-handler-2.1.1-cdh6.3.0.jar