sqoop导出,hive中空值导致的异常

先看问题:

18/09/30 10:52:44 INFO mapreduce.ExportJobBase: Exported 0 records.
18/09/30 10:52:44 ERROR mapreduce.ExportJobBase: Export job failed!
18/09/30 10:52:44 DEBUG util.ClassLoaderStack: Restoring classloader: sun.misc.Launcher$AppClassLoader@6e1567f1
18/09/30 10:52:44 ERROR tool.ExportTool: Error during export:
Export job failed!
at org.apache.sqoop.mapreduce.ExportJobBase.runExport(ExportJobBase.java:445)
at org.apache.sqoop.manager.SqlManager.exportTable(SqlManager.java:931)
at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:80)
at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:99)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
at org.apache.sqoop.Sqoop.main(Sqoop.java:252)

sqoop导出任务不间断出问题 ,后来发现hive中的null值在文件按\N存储,所以导致导出字符异常,参考:
https://stackoverflow.com/questions/30736049/sqoop-export-is-failing-when-i-have-n-as-data

增加选项:
--input-null-string "\\\\N" --input-null-non-string "\\\\N"


-------
感觉在不断探坑。。。
sqoop导出,hive中空值导致的异常

sqoop导出,update-key功能空指针异常问题处理

从sqoop导出mysql,涉及hdfs、hive、cos(腾讯云对象存储)等数据源,sqoop 1.4.6导出 hive表到 mysql,如果需要用到update-key功能(根据主key更新或插入),1.4.6版本实现上有bug,1.4.7版本已经解决。

只能通过hdfs、cos方式导出,注意默认hive数据分隔符是 ctrl+a,所以需要制定分隔符"\0001"

问题:
18/09/29 16:25:42 WARN hcat.SqoopHCatUtilities: No files under /usr/local/service/hive/hcatalog/lib to add to distributed cache for hcatalog job
18/09/29 16:25:42 WARN hcat.SqoopHCatUtilities: No files under /usr/local/service/hive/hcatalog/share/hcatalog/storage-handlers to add to distributed cache for hcatalog job
18/09/29 16:25:42 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.NullPointerException
java.lang.NullPointerException
at org.apache.sqoop.mapreduce.ExportJobBase.getFileType(ExportJobBase.java:127)
at org.apache.sqoop.mapreduce.ExportJobBase.isSequenceFiles(ExportJobBase.java:118)
at org.apache.sqoop.mapreduce.ExportJobBase.inputIsSequenceFiles(ExportJobBase.java:492)
at org.apache.sqoop.mapreduce.JdbcUpdateExportJob.getMapperClass(JdbcUpdateExportJob.java:69)
at org.apache.sqoop.mapreduce.ExportJobBase.configureMapper(ExportJobBase.java:268)
at org.apache.sqoop.mapreduce.ExportJobBase.runExport(ExportJobBase.java:426)
at org.apache.sqoop.manager.MySQLManager.upsertTable(MySQLManager.java:136)
at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:74)
at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:100)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)

解决方案:

sqoop export --connect jdbc:mysql://ip:port/db --table table_name
--username xxx --password xxx
--export-dir hdfs路径or cos路径
--update-key mysql-table-primary-key --update-mode allowinsert
--input-fields-terminated-by "\0001"

sqoop导出,update-key功能空指针异常问题处理

python WindowsError: [Error 145] :解决

执行自动化脚本时,删除文件夹里旧文件报错。如下
C:\ProgramData\Anaconda2\python.exe F:/AutoStartTime/Record_QQLive.py
删除旧文件 -- 开始
Traceback (most recent call last):
File "F:/AutoStartTime/Record_QQLive.py", line 197, in
QQLive = Record_QQLive(serial='cddfc295')
File "F:/AutoStartTime/Record_QQLive.py", line 26, in init
self.test_initEnv()
File "F:/AutoStartTime/Record_QQLive.py", line 92, in test_initEnv
shutil.rmtree(self.pcAppPath)
File "C:\ProgramData\Anaconda2\lib\shutil.py", line 261, in rmtree
rmtree(fullname, ignore_errors, onerror)
File "C:\ProgramData\Anaconda2\lib\shutil.py", line 261, in rmtree
rmtree(fullname, ignore_errors, onerror)
File "C:\ProgramData\Anaconda2\lib\shutil.py", line 270, in rmtree
onerror(os.rmdir, path, sys.exc_info())
File "C:\ProgramData\Anaconda2\lib\shutil.py", line 268, in rmtree
os.rmdir(path)
WindowsError: [Error 145] : u'QQLive\ad\nocache'

原因: 该文件夹处于被打开状态

解决办法: 关闭文件夹后再执行脚本 。so sad~~~低级错误

python WindowsError: [Error 145] :解决

windows10 subsystem for linux adb 连接识别设备

系统:windows10 subsystem for linux ubuntu 16.04

问题:安装了Android-tools 之后 adb 可以执行,但adb devices拉不到设备list

原因: windows 系统的adb version 必须与linux 子系统的adb version保持一致

解决步骤:
1、ubuntu adb 安装
sudo apt-get update && sudo apt-get install android-tools-adb

2、重新下载一个与windows adb version 一样的 linux 版本adb (查看命令 adb version)
wget -O - https://skia.googlesource.com/skia/+archive/cd048d18e0b81338c1a04b9749a00444597df394/platform_tools/android/bin/linux.tar.gz | tar -zxvf - adb

3、覆盖/usr/bin/adb
* sudo mv adb /usr/bin/adb
* sudo chmod +x /usr/bin/adb (如果有执行权限可省略这一步)
* adb version (查看版本是否与windows下的adb 一致)
* adb devices (成功拉取到设备list)

windows10 subsystem for linux adb 连接识别设备

python 自动化uiautomator 报错:ioerror RPC server not started解决办法

在使用最新的VIVO X21手机过程中,自动化脚本报错:

Traceback (most recent call last):
File "main.py", line 75, in
Automated.test_auto_test()
File "main.py", line 22, in test_auto_test
QQLive = Record_QQLive(serial='2973cab8')
File "/mnt/f/AutoStartTime/Record_QQLive.py", line 25, in init
self.test_initEnv()
File "/mnt/f/AutoStartTime/Record_QQLive.py", line 93, in test_initEnv
self.dins.pressHome()
File "/mnt/f/AutoStartTime/androidDrive/deviceInstance.py", line 85, in pressHome
self.ins.pressHome()
File "/mnt/f/AutoStartTime/androidDrive/deviceInstance.py", line 239, in pressHome
self._sys.pressHome()
File "/mnt/f/AutoStartTime/androidDrive/sysActions.py", line 49, in pressHome
self._d.press("home")
File "/mnt/f/AutoStartTime/androidDrive/helper.py", line 140, in call
return self.func(*args, **kwargs)
File "/mnt/f/AutoStartTime/androidDrive/automatorDevice.py", line 233, in _press
return self.server.jsonrpc.pressKey(str(key))
File "/mnt/f/AutoStartTime/androidDrive/automatorServer.py", line 81, in wrapper
server.start(timeout=30)
File "/mnt/f/AutoStartTime/androidDrive/automatorServer.py", line 122, in start
**raise IOError("RPC server not started!")
IOError: RPC server not started!
**

出错原因:
7.0的手机不会自动安装app-uiautomator.apk和app-uiautomator-test.apk,手动装一下就可以了

解决办法如下:
1. adb shell pm uninstall com.github.uiautomator
2. adb shell pm uninstall com.github.uiautomator.test
3. adb install app-uiautomator.apk
4. adb install app-uiautomator-test.apk
5. adb push bundle.jar /data/local/tmp/
6. adb push uiautomator-stub.jar /data/local/tmp/

相关文件下载地址:
https://github.com/jiankehtt/uiautomator/tree/master/uiautomator/libs

python 自动化uiautomator 报错:ioerror RPC server not started解决办法