月度归档： 2020 年 3 月

单机伪分布式zookeeper集群安装

先搭建伪分布式集群，再去搭建真分布式集群。有些的人的电脑内存、性能比较低，所以在搭建真实的一个分布式环境的话，可能会相对来说比较卡，所以两种都会做一下，首先会在单机上搭建一个集群。单机上的集群主要就是它的port，端口号不一致，ip是一样的。称之为伪分布式是因为，当一个节点挂掉之后，整个集群，整个三台zk的节点全部都会挂掉，所以它是一个伪分布式集群。

下载：

$ wget https://archive.apache.org/dist/zookeeper/zookeeper-3.4.14/zookeeper-3.4.14.tar.gz

解压

tar -zxvf zookeeper-3.4.14.tar.gz

移动到~/app目录下

$ mv zookeeper-3.4.14 ~/app

将解压出来的zookeeper文件夹复制出来三个副本 分别是zk1 zk2 和 zk3 

[hadoop@hadoop000 app]$ cp zookeeper-3.4.11/ zk1 -rf
[hadoop@hadoop000 app]$ cp zookeeper-3.4.11/ zk2 -rf
[hadoop@hadoop000 app]$ cp zookeeper-3.4.11/ zk3 -rf

为每个副本中的dataDir中创建一个myid, 其中zk1的myid为1, zk2的myid为2, zk3的myid为3 

[hadoop@hadoop000 app]$ echo 1 > zk1/dataDir/myid
[hadoop@hadoop000 app]$ echo 2 > zk2/dataDir/myid
[hadoop@hadoop000 app]$ echo 3 > zk3/dataDir/myid

分别修改每个副本中的conf/zoo.cfg文件 

[hadoop@hadoop000 app]$ vi zk1/conf/zoo.cfg
tickTime=2000				# 单个心跳的毫秒数
initLimit=10				# 同步阶段初始化所需的心跳数
syncLimit=5				# 请求和响应所需的心跳数
dataDir=/home/hadoop/app/zk1/dataDir		# 快照数据存放位置 - 需对应修改
dataLogDir=/home/hadoop/app/zk1/dataLogDir	# 事务日志存放位置 - 需对应修改
clientPort=2191				# 客户端连接端口 - 需对应修改, 不能重复
# 将所有命令添加到白名单中
4lw.commands.whitelist=*
	
# 配置zookeeper服务器的 [主机名或主机ID]:[同步端口(唯一)]:[选举端口(唯一)]
server.1=localhost:2891:3891
server.2=localhost:2892:3892
server.3=localhost:2893:3893

[hadoop@hadoop000 app]$ vi zk2/conf/zoo.cfg
tickTime=2000				# 单个心跳的毫秒数
initLimit=10				# 同步阶段初始化所需的心跳数
syncLimit=5				# 请求和响应所需的心跳数
dataDir=/home/hadoop/app/zk2/dataDir		# 快照数据存放位置 - 需对应修改
dataLogDir=/home/hadoop/app/zk2/dataLogDir	# 事务日志存放位置 - 需对应修改
clientPort=2192			# 客户端连接端口 - 需对应修改, 不能重复
# 将所有命令添加到白名单中
4lw.commands.whitelist=*	
# 配置zookeeper服务器的 [主机名或主机ID]:[同步端口(唯一)]:[选举端口(唯一)]
server.1=localhost:2891:3891
server.2=localhost:2892:3892
server.3=localhost:2893:3893
[hadoop@hadoop000 app]$ vi zk3/conf/zoo.cfg
tickTime=2000				# 单个心跳的毫秒数
initLimit=10				# 同步阶段初始化所需的心跳数
syncLimit=5				# 请求和响应所需的心跳数
dataDir=/home/hadoop/app/zk3/dataDir		# 快照数据存放位置 - 需对应修改
dataLogDir=/home/hadoop/app/zk3/dataLogDir	# 事务日志存放位置 - 需对应修改
clientPort=2193				# 客户端连接端口 - 需对应修改, 不能重复
# 将所有命令添加到白名单中
4lw.commands.whitelist=*
# 配置zookeeper服务器的 [主机名或主机ID]:[同步端口(唯一)]:[选举端口(唯一)]
server.1=localhost:2891:3891
server.2=localhost:2892:3892
server.3=localhost:2893:3893

修改好之后, 分别重启三个副本的zk服务 (如果已经start则先stop) 

[hadoop@hadoop000 app]$ ./zk1/bin/zkServer.sh start
[hadoop@hadoop000 app]$ ./zk2/bin/zkServer.sh start
[hadoop@hadoop000 app]$ ./zk3/bin/zkServer.sh start

使用zk客户端连接zk1 

[hadoop@hadoop000 app]$ zkCli.sh -server localhost:2191
Connecting to localhost:2191
2020-03-01 18:23:45,704 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.11-37e277162d567b55a07d1755f0b31c32e93c01a0, built on 11/01/2017 18:06 GMT
2020-03-01 18:23:45,710 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=hadoop000
2020-03-01 18:23:45,710 [myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.8.0_211
2020-03-01 18:23:45,712 [myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2020-03-01 18:23:45,713 [myid:] - INFO  [main:Environment@100] - Client environment:java.home=/home/hadoop/app/jdk1.8.0_211/jre
2020-03-01 18:23:45,713 [myid:] - INFO  [main:Environment@100] - Client environment:java.class.path=/home/hadoop/app/zookeeper-3.4.11/bin/../build/classes:/home/hadoop/app/zookeeper-3.4.11/bin/../build/lib/*.jar:/home/hadoop/app/zookeeper-3.4.11/bin/../lib/slf4j-log4j12-1.6.1.jar:/home/hadoop/app/zookeeper-3.4.11/bin/../lib/slf4j-api-1.6.1.jar:/home/hadoop/app/zookeeper-3.4.11/bin/../lib/netty-3.10.5.Final.jar:/home/hadoop/app/zookeeper-3.4.11/bin/../lib/log4j-1.2.16.jar:/home/hadoop/app/zookeeper-3.4.11/bin/../lib/jline-0.9.94.jar:/home/hadoop/app/zookeeper-3.4.11/bin/../lib/audience-annotations-0.5.0.jar:/home/hadoop/app/zookeeper-3.4.11/bin/../zookeeper-3.4.11.jar:/home/hadoop/app/zookeeper-3.4.11/bin/../src/java/lib/*.jar:/home/hadoop/app/zookeeper-3.4.11/bin/../conf:
2020-03-01 18:23:45,713 [myid:] - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2020-03-01 18:23:45,713 [myid:] - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2020-03-01 18:23:45,713 [myid:] - INFO  [main:Environment@100] - Client environment:java.compiler=<NA>
2020-03-01 18:23:45,713 [myid:] - INFO  [main:Environment@100] - Client environment:os.name=Linux
2020-03-01 18:23:45,713 [myid:] - INFO  [main:Environment@100] - Client environment:os.arch=amd64
2020-03-01 18:23:45,713 [myid:] - INFO  [main:Environment@100] - Client environment:os.version=3.10.0-514.el7.x86_64
2020-03-01 18:23:45,714 [myid:] - INFO  [main:Environment@100] - Client environment:user.name=hadoop
2020-03-01 18:23:45,714 [myid:] - INFO  [main:Environment@100] - Client environment:user.home=/home/hadoop
2020-03-01 18:23:45,714 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/home/hadoop/app
2020-03-01 18:23:45,715 [myid:] - INFO  [main:ZooKeeper@441] - Initiating client connection, connectString=localhost:2191 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@25f38edc
Welcome to ZooKeeper!
JLine support is enabled
2020-03-01 18:23:45,789 [myid:] - INFO  [main-SendThread(localhost:2191):ClientCnxn$SendThread@1035] - Opening socket connection to server localhost/0:0:0:0:0:0:0:1%1:2191. Will not attempt to authenticate using SASL (unknown error)
2020-03-01 18:23:45,899 [myid:] - INFO  [main-SendThread(localhost:2191):ClientCnxn$SendThread@877] - Socket connection established to localhost/0:0:0:0:0:0:0:1%1:2191, initiating session
2020-03-01 18:23:45,974 [myid:] - INFO  [main-SendThread(localhost:2191):ClientCnxn$SendThread@1302] - Session establishment complete on server localhost/0:0:0:0:0:0:0:1%1:2191, sessionid = 0x1000069c5b30000, negotiated timeout = 30000
WATCHER::
WatchedEvent state:SyncConnected type:None path:null

列出zk1根目录下节点信息 

[zk: localhost:2191(CONNECTED) 0] ls /
[zookeeper]

创建data节点 并保存数据2191 

[zk: localhost:2191(CONNECTED) 1] create /data 2191
Created /data

查看data节点的数据

[zk: localhost:2191(CONNECTED) 0] get /data
2191
cZxid = 0x100000002
ctime = Sun Mar 01 18:24:15 CST 2020
mZxid = 0x100000002
mtime = Sun Mar 01 18:24:15 CST 2020
pZxid = 0x100000002
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 4
numChildren = 0

退出zk1客户端 

[zk: localhost:2191(CONNECTED) 3] quitQuitting...

连接zk2客户端 

[hadoop@hadoop000 app]$ zkCli.sh -server localhost:2192

查看zk2客户端的data节点的数 (zk1同步过来的数据) 

[zk: localhost:2192(CONNECTED) 0] get /data
2191

连接zk3客户端 

[hadoop@hadoop000 app]$ zkCli.sh -server localhost:2193

查看zk3的data节点的数 (同步过来的数据) 

[zk: localhost:2193(CONNECTED) 0] get /data
2191

可见对任何一个服务器节点进行操作, 都会将结果同步到其他服务器的节点上!

查看进程 

[hadoop@hadoop000 app]$ jps
3780 QuorumPeerMain
3670 QuorumPeerMain
3718 QuorumPeerMain
3995 Jps

查看服务器状态 

[hadoop@hadoop000 app]$ ./zk1/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /home/hadoop/app/zk1/bin/../conf/zoo.cfg
Mode: follower
[hadoop@hadoop000 app]$ ./zk2/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /home/hadoop/app/zk2/bin/../conf/zoo.cfg
Mode: leader
[hadoop@hadoop000 app]$ ./zk3/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /home/hadoop/app/zk3/bin/../conf/zoo.cfg
Mode: follower

可见zk2是选举出来的leader
而zk1和zk3是其追随者 follower

如果服务状态显示服务没有启动, 可以去相应的目录下寻找服务器日志, 分析并解决问题.

Views: 71

Hive(1.2.2) Installation

MySQL的Hive安装

安装MySQL(可以在不同机器上, 但网络要可以互访)

在里面创建一个允许远程访问的MySQL账号

CREATE USER 'hive'@'%' IDENTIFIED BY 'hive1234';
GRANT ALL PRIVILEGES ON hive.* TO 'hive'@'%' WITH GRANT OPTION;
CREATE USER 'hive'@'localhost' IDENTIFIED BY 'hive1234';
GRANT ALL PRIVILEGES ON hive.* TO 'hive'@'localhost' WITH GRANT OPTION;
  • 在虚拟机测试能否远程连接MySQL服务
    • 有MySQL客户端的情况下
      • mysql -uhive -h <mysql所在机器的ip> -p
      • 输出密码回车, 如果能进入mysql>命令行界面说明连接成功
    • 没有MySQL客户端也可以直接ping一下mysql所在机器的ip

下载hive, 这里选择1.2.2版本

解压到 ~/app下, 可以视你的情况更改, 但要保证有读写权限,

编辑~/.bash_profile 添加环境变量, 红色高亮处要根据你的实际情况修改

export HIVE_HOME=/home/hadoop/app/hive-1.2.2(替换为hive实际解压位置)

export PATH=$HIVE_HOME/bin:$PATH

输入source ~/.bash_profile 使其生效

终端输入hiv按tab键, 如果能自动补全为hive说明环境变量ok

进入hive解压目录的conf文件夹下面, hive-env.sh 和 hive-site.xml 两个文件

    • 创建 hive-env.sh, 并添加下面配置
HADOOP_HOME=/home/hadoop/app/hadoop-2.6.0-cdh5.15.1(换成你hadoop所在目录)
    • 创建hive-site.xml, 参照示例修改
 <?xml version="1.0"?>
 <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
 <configuration>
 <property>
   <name>javax.jdo.option.ConnectionURL</name>
     <value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true&amp;useSSL=false(如果MySQL和hive在一个机器上可以使用localhost, 否则需要填写MySQL所在机器的实际IP)</value>
 </property>
 <property>
   <name>javax.jdo.option.ConnectionDriverName</name>
   <value>com.mysql.jdbc.Driver</value>
 </property>
 <property>
   <name>javax.jdo.option.ConnectionUserName</name>
   <value>hive(换成你实际使用的MySQL账户)</value>
 </property>
 <property>
   <name>javax.jdo.option.ConnectionPassword</name>
   <value>hive1234(换成你使用MySQL账户的实际密码)</value>
 </property>
 </configuration>

下载mysql-connector--java-5.1.xx.bin.jar 放在 ${HIVE_HOME}/lib 下面

启动dfs 和 yarn

 [hadoop@hadoop000 conf]$ jps
 11602 NameNode
 12212 ResourceManager
 11863 SecondaryNameNode
 11706 DataNode
 12315 NodeManager
 48207 Jps

 等待几分钟后, 避开安全模式, 启动 hive客户端

[hadoop@hadoop000 ~]$ hive
PS:
如果MysQL服务器部署在其他机器上, 则URL中要改成IP或主机名的访问方式
如: jdbc:mysql://192.168.1.142:3306/hive?createDatabaseIfNotExist=true&amp;useSSL=false
这种情况下所使用的账户必须要开启远程访问权限 hive@%

Hive快速入门

hive sql 语法 https://cwiki.apache.org/confluence/display/Hive/LanguageManual

hive的hql语句和sql极其类似

[hadoop@hadoop000 ~]$ hive
which: no hbase in (/home/hadoop/app/hive-1.1.0-cdh5.15.1/bin:/home/hadoop/app/hadoop-2.6.0-cdh5.15.1/bin:/home/hadoop/app/hadoop-2.6.0-cdh5.15.1/sbin:/home/hadoop/app/jdk1.8.0_91/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/hadoop/.local/bin:/home/hadoop/bin)

Logging initialized using configuration in jar:file:/home/hadoop/app/hive-1.1.0-cdh5.15.1/lib/hive-common-1.1.0-cdh5.15.1.jar!/hive-log4j.properties
WARNING: Hive CLI is deprecated and migration to Beeline is recommended.
hive> 
hive> show databases;
OK
default
Time taken: 22.65 seconds, Fetched: 1 row(s)
hive> create database hive;
OK
Time taken: 0.372 seconds
hive> use hive;
OK
Time taken: 0.069 seconds
hive> show tables;
OK
Time taken: 0.074 seconds
hive>

这时登录hive用户登录mysql中查看, 会发现自动创建了hive数据库, 并且里面出现了26张表(这些就是用来保存元数据的)

mysql> use hive;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> show tables;
+---------------------------+
| Tables_in_hive            |
+---------------------------+
| BUCKETING_COLS            |
| CDS                       |
| COLUMNS_V2                |
| DATABASE_PARAMS           |
| DBS                       |
| FUNCS                     |
| FUNC_RU                   |
| GLOBAL_PRIVS              |
| PARTITIONS                |
| PARTITION_KEYS            |
| PARTITION_KEY_VALS        |
| PARTITION_PARAMS          |
| PART_COL_STATS            |
| ROLES                     |
| SDS                       |
| SD_PARAMS                 |
| SEQUENCE_TABLE            |
| SERDES                    |
| SERDE_PARAMS              |
| SKEWED_COL_NAMES          |
| SKEWED_COL_VALUE_LOC_MAP  |
| SKEWED_STRING_LIST        |
| SKEWED_STRING_LIST_VALUES |
| SKEWED_VALUES             |
| SORT_COLS                 |
| TABLE_PARAMS              |
| TAB_COL_STATS             |
| TBLS                      |
| VERSION                   |
+---------------------------+
29 rows in set (0.00 sec)

其中DBS那张表中保存的就是数据库的相关信息

mysql> select * from DBS \G
*************************** 1. row ***************************
          DB_ID: 1
           DESC: Default Hive database
DB_LOCATION_URI: hdfs://hadoop000:8020/user/hive/warehouse
           NAME: default
     OWNER_NAME: public
     OWNER_TYPE: ROLE
*************************** 2. row ***************************
          DB_ID: 2
           DESC: NULL
DB_LOCATION_URI: hdfs://hadoop000:8020/user/hive/warehouse/hive.db
           NAME: hive
     OWNER_NAME: hadoop
     OWNER_TYPE: USER
2 rows in set (0.00 sec)

Views: 24