Sqoop

Sqoop:将关系数据库(oracle、mysql、postgresql等)数据与hadoop数据进行转换的工具
官网:http://sqoop.apache.org/
版本:(两个版本完全不兼容,sqoop1使用最多)
sqoop1:1.4.x
sqoop2:1.99.x
同类产品
DataX:阿里*数据交换工具
sqoop架构非常简单,是hadoop生态系统的架构最简单的框架。
sqoop1由client端直接接入hadoop,任务通过解析生成对应的maprecue执行
Sqoop导入:
Sqoop导出:
Sqoop安装步骤:
1、解压
2、配置环境变量
export SQOOP_HOME=/XX/sqoop.xx
source /etc/profile
3、添加数据库驱动包
cp mysql-connector-java-5.1.10.jar /sqoop-install-path/lib
4、重命名配置文件
mv sqoop-env-template.sh sqoop-env.sh
5、修改配置configure-sqoop
去掉未安装服务相关内容;例如(HBase、HCatalog、Accumulo):
#if [ ! -d “${HBASE_HOME}” ]; then

echo “Error: $HBASE_HOME does not exist!”

echo ‘Please set $HBASE_HOME to the root of your HBase installation.’

exit 1

6、测试
sqoop version
sqoop list-databases -connect jdbc:mysql://node3:3306/ -username root -password 123
sqoop工具:
SqoopSqoop导出工具export:
Sqoopsqoop:
导入: mysql数据库中表导入到hadoop

import
–connect
jdbc:mysql://hadoop1:3306/test
–username
root
–password
123
–as-textfile
–columns
id,name,msg
–table
psn
–delete-target-dir
–target-dir
/sqoop/data
-m
1

命令:
sqoop --options-file sqoop1


导入: mysql数据库中表导入到hive

import
–connect
jdbc:mysql://hadoop1/test
–username
root
–password
123
–as-textfile
–query
‘select id, name, msg from psn where id like “1%” and $CONDITIONS’
–delete-target-dir
–target-dir
/sqoop/tmp
-m
1
–hive-home
/home/hive-1.2.1
–hive-import
–create-hive-table
–hive-table
t_test


导出: hadoop数据导出到mysql

export
–connect
jdbc:mysql://node1/test
–username
root
–password
123
-m
1
–columns
id,name,msg
–export-dir
/sqoop/data
–table
h_psn