sqoop 安装与配置使用
2021/8/20 8:05:37
本文主要是介绍sqoop 安装与配置使用,对大家解决编程问题具有一定的参考价值,需要的程序猿们随着小编来一起学习吧!
sqoop 安装与配置使用
hive 与 mysql/pgsql 数据传递
sqoop 安装
安装包 上传至 hive-server.
$ docker cp mysql-connector-java-5.1.28.jar dc387ff5c56d:/opt/mysql-connector-java-5.1.28.jar $ docker cp sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz dc387ff5c56d:/opt/sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz
进入 hive-server 安装 sqoop
$ docker exec -it dc387ff5c56d /bin/bash # tar -zxvf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz # rm sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz # mv sqoop-1.4.6.bin__hadoop-2.0.4-alpha/ sqoop # mv mysql-connector-java-5.1.28.jar sqoop/lib/mysql-connector-java-5.1.28.jar # cp sqoop/conf/sqoop-env-template.sh sqoop/conf/sqoop-env.sh # vim sqoop/conf/sqoop-env.sh #Set the path to where bin/hive is available export HIVE_HOME=/opt/hive/
sqoop 将sqoop 数据导入
# /opt/sqoop/bin/sqoop list-databases --connect jdbc:mysql://192.168.20.204:3306/ --username ad_user --password 'dl_ad!@#123' Warning: /opt/sqoop/bin/../../hbase does not exist! HBase imports will fail. Please set $HBASE_HOME to the root of your HBase installation. Warning: /opt/sqoop/bin/../../hcatalog does not exist! HCatalog jobs will fail. Please set $HCAT_HOME to the root of your HCatalog installation. Warning: /opt/sqoop/bin/../../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. Warning: /opt/sqoop/bin/../../zookeeper does not exist! Accumulo imports will fail. Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation. 21/07/30 10:24:23 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6 21/07/30 10:24:23 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 21/07/30 10:24:23 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. addata dl_ad dl_college e_statistics i_commodity i_marketing i_member i_operation i_order information_schema log mysql performance_schema test
在 hive 中建立 mysql 对应需要同步的数据库
[work@Lan-Kvm-20157-Reptile2 ~]$ docker exec -it dc387ff5c56d /bin/bash root@dc387ff5c56d:/opt# hive hive> show databases; OK default Time taken: 1.081 seconds, Fetched: 1 row(s) # hive 创建对应数据库 hive> create database i_member; OK Time taken: 0.155 seconds hive> create database i_order; OK Time taken: 0.074 seconds hive> create database i_commodity; OK Time taken: 0.074 seconds hive> create database i_marketing; OK Time taken: 0.058 seconds hive> create database i_operation; OK Time taken: 0.069 seconds # 导入 mysql 数据库
数据导入
$ /opt/sqoop/bin/sqoop import \ --connect jdbc:mysql://192.168.20.204:3306/i_member \ --username ad_user \ --password 'dl_ad!@#123' \ --table t_member \ --num-mappers 1 \ --hive-import \ --fields-terminated-by "\t" \ --hive-overwrite \ --hive-database i_member \ --hive-table t_member 21/08/01 08:48:01 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 3.1657 seconds (0 bytes/sec) 21/08/01 08:48:01 INFO mapreduce.ImportJobBase: Retrieved 1000 records. 21/08/01 08:48:01 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `t_member` AS t LIMIT 1 21/08/01 08:48:01 WARN hive.TableDefWriter: Column create_time had to be cast to a less precise type in Hive 21/08/01 08:48:01 WARN hive.TableDefWriter: Column update_time had to be cast to a less precise type in Hive 21/08/01 08:48:01 INFO hive.HiveImport: Loading uploaded data into Hive 21/08/01 08:48:09 INFO hive.HiveImport: SLF4J: Class path contains multiple SLF4J bindings. 21/08/01 08:48:09 INFO hive.HiveImport: SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] 21/08/01 08:48:09 INFO hive.HiveImport: SLF4J: Found binding in [jar:file:/opt/hadoop-2.7.1/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] 21/08/01 08:48:09 INFO hive.HiveImport: SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 21/08/01 08:48:10 INFO hive.HiveImport: 21/08/01 08:48:10 INFO hive.HiveImport: Logging initialized using configuration in file:/opt/hive/conf/hive-log4j2.properties Async: true 21/08/01 08:48:12 INFO hive.HiveImport: OK 21/08/01 08:48:12 INFO hive.HiveImport: Time taken: 1.583 seconds 21/08/01 08:48:13 INFO hive.HiveImport: Loading data to table i_member.t_member 21/08/01 08:48:15 INFO hive.HiveImport: OK 21/08/01 08:48:15 INFO hive.HiveImport: Time taken: 2.832 seconds 21/08/01 08:48:16 INFO hive.HiveImport: Hive import complete. # 此时访问 http://192.168.20.157:50070/explorer.html#/user/hive/warehouse/i_member.db 就可以查看到 i_member 数据库中 t_member 表
查询数据验证
hive> use i_member; OK Time taken: 0.04 seconds hive> show tables; OK t_member Time taken: 0.064 seconds, Fetched: 1 row(s) hive> select * from t_member; OK 1 bywind1 password 2 15321761517 1 2 k8fy5od-tunhgvlx 1 2019-11-10 09:00:00.0 2019-11-23 10:22:01.0 2 bywind2 password 1 15321761517 2 1 u694h5d2r0j-znib 1 2019-11-10 09:00:00.0 2019-11-29 10:22:01.0 3 bywind3 password 2 15321761517 3 1 _5836va1ck7hx9mt 1 2019-11-10 09:00:00.0 2019-11-22 10:22:01.0 4 bywind4 password 2 15321761517 4 3 t8a2wf_o3k0zdeiu 1 2019-11-10 09:00:00.0 2019-11-27 10:22:01.0 5 bywind5 password 2 15321761517 5 1 -g0m6qku4sfv97yn 1 2019-11-10 09:00:00.0 2019-11-23 10:22:01.0
这篇关于sqoop 安装与配置使用的文章就介绍到这儿,希望我们推荐的文章对大家有所帮助,也希望大家多多支持为之网!
- 2024-10-01基于Python+Vue开发的医院门诊预约挂号系统
- 2024-10-01基于Python+Vue开发的旅游景区管理系统
- 2024-10-01RestfulAPI入门指南:打造简单易懂的API接口
- 2024-10-01初学者指南:了解和使用Server Action
- 2024-10-01Server Component入门指南:搭建与配置详解
- 2024-10-01React 中使用 useRequest 实现数据请求
- 2024-10-01使用 golang 将ETH账户的资产平均分散到其他账户
- 2024-10-01JWT用户校验课程:从入门到实践
- 2024-10-01Server Component课程入门指南
- 2024-09-30Dnd-Kit学习:新手快速入门指南