sqoop导出,update-key功能空指针异常问题处理

从sqoop导出mysql,涉及hdfs、hive、cos(腾讯云对象存储)等数据源,sqoop 1.4.6导出 hive表到 mysql,如果需要用到update-key功能(根据主key更新或插入),1.4.6版本实现上有bug,1.4.7版本已经解决。

只能通过hdfs、cos方式导出,注意默认hive数据分隔符是 ctrl+a,所以需要制定分隔符"\0001"

问题:
18/09/29 16:25:42 WARN hcat.SqoopHCatUtilities: No files under /usr/local/service/hive/hcatalog/lib to add to distributed cache for hcatalog job
18/09/29 16:25:42 WARN hcat.SqoopHCatUtilities: No files under /usr/local/service/hive/hcatalog/share/hcatalog/storage-handlers to add to distributed cache for hcatalog job
18/09/29 16:25:42 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.NullPointerException
java.lang.NullPointerException
at org.apache.sqoop.mapreduce.ExportJobBase.getFileType(ExportJobBase.java:127)
at org.apache.sqoop.mapreduce.ExportJobBase.isSequenceFiles(ExportJobBase.java:118)
at org.apache.sqoop.mapreduce.ExportJobBase.inputIsSequenceFiles(ExportJobBase.java:492)
at org.apache.sqoop.mapreduce.JdbcUpdateExportJob.getMapperClass(JdbcUpdateExportJob.java:69)
at org.apache.sqoop.mapreduce.ExportJobBase.configureMapper(ExportJobBase.java:268)
at org.apache.sqoop.mapreduce.ExportJobBase.runExport(ExportJobBase.java:426)
at org.apache.sqoop.manager.MySQLManager.upsertTable(MySQLManager.java:136)
at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:74)
at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:100)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)

解决方案:

sqoop export --connect jdbc:mysql://ip:port/db --table table_name
--username xxx --password xxx
--export-dir hdfs路径or cos路径
--update-key mysql-table-primary-key --update-mode allowinsert
--input-fields-terminated-by "\0001"

sqoop导出,update-key功能空指针异常问题处理