07 Flume的安装与配置

安装

  1. 下载

    http://flume.apache.org/download.html

    http://archive.apache.org/dist/flume/stable/

    wget http://archive.apache.org/dist/flume/stable/apache-flume-1.9.0-bin.tar.gz

    这里使用最新的 apache-flume-1.9.0版本

  2. 解压安装

    tar zxvf apache-flume-1.9.0-bin.tar.gz -C /opt/pkg/
  3. 改目录名

    mv apache-flume-1.9.0-bin/ flume
  4. 配置环境变量,并让环境变量生效

    # FLUME 1.9.0
    export FLUME_HOME=/opt/pkg/flume
    export PATH=$FLUME_HOME/bin:$PATH
  5. 修改conf/flume-env.sh,配置JDK路径(该文件事先是不存在的,需要复制一份)
    复制:

    cp flume-env.template.sh flume-env.sh

    编辑文件,并设置如下内容:

    #设置JAVA_HOME:
    export JAVA_HOME = /opt/pkg/java         
    #修改默认的内存:  
    export JAVA_OPTS="-Xms1024m -Xmx1024m -Xss256k -Xmn2g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit"   
  6. 将hadoop-3.1.4安装路径下的依赖的jar软链接到flume-1.9.0/lib下:

    $ cd /opt/pkg/flume/lib
    $ ln -s /opt/pkg/hadoop/share/hadoop/common/hadoop-common-3.1.4.jar ./
    $ ln -s /opt/pkg/hadoop/share/hadoop/common/lib/commons-configuration2-2.1.1.jar ./
    $ ln -s /opt/pkg/hadoop/share/hadoop/common/lib/hadoop-auth-3.1.4.jar ./      
    $ ln -s /opt/pkg/hadoop/share/hadoop/common/lib/htrace-core4-4.1.0-incubating.jar ./
    $ ln -s /opt/pkg/hadoop/share/hadoop/common/lib/commons-io-2.5.jar ./         
    $ ln -s /opt/pkg/hadoop/share/hadoop/hdfs/hadoop-hdfs-3.1.4.jar ./
    

测试

  1. 验证

    bin/flume-ng version
    
    flume 1.9.0
    Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git
    Revision: d4fcab4f501d41597bc616921329a4339f73585e
    Compiled by fszabo on Mon Dec 17 20:45:25 CET 2018
    From source with checksum 35db629a3bda49d23e9b3690c80737f9
  2. 配置Flume HDFS Sink:
    在flume的conf目录新建一个log2hdfs.conf
    添加如下内容:

    [hadoop@hadoop100 conf]$ vi log2hdfs.conf
    
    # define the agent
    a1.sources=r1
    a1.channels=c1
    a1.sinks=k1
    
    # define the source
    #上传目录类型
    a1.sources.r1.type=spooldir
    a1.sources.r1.spoolDir=/tmp/flume-logs
    #定义自滚动日志完成后的后缀名
    a1.sources.r1.fileSuffix=.FINISHED
    #根据每行文本内容的大小自定义最大长度4096=4k
    a1.sources.r1.deserializer.maxLineLength=4096
    
    # define the sink
    a1.sinks.k1.type = hdfs
    #上传的文件保存在hdfs的/flume/logs目录下
    a1.sinks.k1.hdfs.path = hdfs://hadoop100:8020/flume/logs/%y-%m-%d/
    a1.sinks.k1.hdfs.filePrefix=access_log
    a1.sinks.k1.hdfs.fileSufix=.log
    a1.sinks.k1.hdfs.batchSize=1000
    a1.sinks.k1.hdfs.fileType = DataStream
    a1.sinks.k1.hdfs.writeFormat= Text
    # roll 滚动规则:按照数据块128M大小来控制文件的写入,与滚动相关其他的都设置成0
    #为了演示,这里设置成500k写入一次
    a1.sinks.k1.hdfs.rollSize= 512000
    a1.sinks.k1.hdfs.rollCount=0
    a1.sinks.k1.hdfs.rollInteval=0
    #控制生成目录的规则:一般是一天或者一周或者一个月一次,这里为了演示设置10秒
    a1.sinks.k1.hdfs.round=true
    a1.sinks.k1.hdfs.roundValue=10
    a1.sinks.k1.hdfs.roundUnit= second
    #是否使用本地时间
    a1.sinks.k1.hdfs.useLocalTimeStamp=true
    
    #define the channel
    a1.channels.c1.type = memory
    #自定义event的条数
    a1.channels.c1.capacity = 500000
    #flume事务控制所需要的缓存容量1000条event
    a1.channels.c1.transactionCapacity = 1000
    
    #source channel sink cooperation
    a1.sources.r1.channels = c1
    a1.sinks.k1.channel = c1
    

    注意:

    • a1.sources.r1.spoolDir目录如果不存在需要先创建
    • a1.sinks.k1.hdfs.path目录会自动创建
      • 这里的路径是`hdfs://hadoop100:8020/flume/logs/yy-mm-dd/
      • 也就是每天的数据都会产生滚动日志。
      • 实际应该是按天或者按周、按月来生成滚动日志。
  3. 启动flume

  • 准备

    创建/tmp/flumn-logs, 并分配权限

     sudo mkdir -p  /tmp/flume-logs
  • 启动
    执行如下命令进行启动:

     flume-ng agent --conf ./conf/ -f ./conf/flume-hdfs.conf --name a1 -Dflume.root.logger=INFO,console
  • 测试

     # 每次执行追加一行文字
     echo xxxxxxxxxxxx >> /tmp/flume-logs/access_log123.log
  • 到Hadoop的控制台http://hadoop100:9870/查看 hdfs:hadoop100:9870/flume/logs/下有没有数据生成:

    image-20210110175013827

异常: 日志收集失败,报错:

2021-03-09 22:38:50,834 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:459)] process failed
java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V

原因: flume/lib/guava-xxx.jar 和 hadoop自带的jar包发生冲突

解决: 将flume/lib下的guava包删除或者改名, 只保留hadoop的版本即可

Views: 138

06 Centos7的MySQL安装

卸载mariaDb

mysql被oracle收购后为了防止mysql有可能变成闭源,因此mysql创始人maria就开源做了一个mariaDb, centos7是自带使用了这个数据库的.因此安装mysql之前,应当首先卸载mariadb数据库:

$ sudo yum list installed | grep mariadb    #检查mariadb是否已安装

$ sudo yum list installed | grep mariadb
mariadb-libs.x86_64                     1:5.5.56-2.el7                 @anaconda

$ sudo yum -y remove mariadb*    #全部卸载

同理如果是已经安装过其他版本的MySQL,安装新的版本之前也需要按照如上方法进行卸载。

安装MySQL

下载mysql的YUM源: https://dev.mysql.com/downloads/repo/yum/

YUM源的官方安装说明:https://dev.mysql.com/doc/mysql-yum-repo-quick-guide/en/

下载源mysql的YUM源,官网提供的是最新的MySQL8.0的源,内含其他较低版本的源

wget -P /home/hadoop/softwares https://dev.mysql.com/get/mysql80-community-release-el7-3.noarch.rpm  --no-check-certificate

由于我们是下载到/home/hadoop/softwares目录下,所以先切换到该目录下:

cd /home/hadoop/softwares

安装mysql的YUM源:

sudo rpm -ivh mysql80-community-release-el7-3.noarch.rpm

检查mysql的YUM源是否安装成功:

sudo yum repolist enabled | grep mysql

mysql-connectors-community/x86_64       MySQL Connectors Community           175
mysql-tools-community/x86_64            MySQL Tools Community                120
mysql80-community/x86_64                MySQL 8.0 Community Server           211

选择要启用的mysql版本

查看源里包含的mysql版本,执行:

sudo yum repolist all | grep mysql

mysql-cluster-7.5-community/x86_64 MySQL Cluster 7.5 Community      禁用
mysql-cluster-7.5-community-source MySQL Cluster 7.5 Community - So 禁用
mysql-cluster-7.6-community/x86_64 MySQL Cluster 7.6 Community      禁用
mysql-cluster-7.6-community-source MySQL Cluster 7.6 Community - So 禁用
mysql-cluster-8.0-community/x86_64 MySQL Cluster 8.0 Community      禁用
mysql-cluster-8.0-community-source MySQL Cluster 8.0 Community - So 禁用
mysql-connectors-community/x86_64  MySQL Connectors Community       启用:    175
mysql-connectors-community-source  MySQL Connectors Community - Sou 禁用
mysql-tools-community/x86_64       MySQL Tools Community            启用:    120
mysql-tools-community-source       MySQL Tools Community - Source   禁用
mysql-tools-preview/x86_64         MySQL Tools Preview              禁用
mysql-tools-preview-source         MySQL Tools Preview - Source     禁用
mysql55-community/x86_64           MySQL 5.5 Community Server       禁用
mysql55-community-source           MySQL 5.5 Community Server - Sou 禁用
mysql56-community/x86_64           MySQL 5.6 Community Server       禁用
mysql56-community-source           MySQL 5.6 Community Server - Sou 禁用
mysql57-community/x86_64           MySQL 5.7 Community Server       禁用
mysql57-community-source           MySQL 5.7 Community Server - Sou 禁用
mysql80-community/x86_64           MySQL 8.0 Community Server       启用:    211
mysql80-community-source           MySQL 8.0 Community Server - Sou 禁用

可以通过类似下面的语句来启动和禁用某些版本,比如这里是禁用默认启用的MySQL8.0的源,转而启用我们需要安装的MySQL5.7的源。

或者通过修改/etc/yum.repos.d/mysql-community.repo文件,改变默认安装的mysql版本。比如要安装5.7版本,将8.0源的enabled=1改成enabled=0,然后再将5.7源的enabled=0改成enabled=1即可。

sudo yum-config-manager --enable mysql57-community
sudo yum-config-manager --disable mysql80-community

要使用yum-config-manager命令, 需要先安装yum-utils: sudo yum install yum-utils

注意: 同一时间只允许enable一个MySQL版本。

如果只需要安装MySQL5.7的版本,也可以不看上面,直接本地安装指定的远程MySQL5.7源

yum localinstall https://dev.mysql.com/get/mysql57-community-release-el7-9.noarch.rpm

查看当前的启用的 MySQL 版本:

yum repolist enabled | grep mysql

mysql-connectors-community/x86_64       MySQL Connectors Community           175
mysql-tools-community/x86_64            MySQL Tools Community                120
mysql57-community/x86_64                MySQL 5.7 Community Server           464

安装MySQL

sudo yum install -y mysql-community-server

已加载插件:fastestmirror
mysql-connectors-community                                             | 2.6 kB  00:00:00     
mysql-tools-community                                                  | 2.6 kB  00:00:00     
mysql57-community                                                      | 2.6 kB  00:00:00     
mysql57-community/x86_64/prima FAILED                                          
http://repo.mysql.com/yum/mysql-5.7-community/el/7/x86_64/repodata/cae8c564a11e5fca11dbe958c273358f206f1bba-primary.sqlite.bz2: [Errno 14] curl#7 - "Failed connect to repo.mysql.com:80; Connection refused"
正在尝试其它镜像。
mysql57-community/x86_64/primary_db                                    | 247 kB  00:00:00     
Loading mirror speeds from cached hostfile
 * base: mirrors.aliyun.com
 * extras: mirrors.aliyun.com
 * updates: mirrors.aliyun.com
正在解决依赖关系
--> 正在检查事务
---> 软件包 mysql-community-server.x86_64.0.5.7.32-1.el7 将被 安装
--> 正在处理依赖关系 mysql-community-common(x86-64) = 5.7.32-1.el7,它被软件包 mysql-community-server-5.7.32-1.el7.x86_64 需要
--> 正在处理依赖关系 mysql-community-client(x86-64) >= 5.7.9,它被软件包 mysql-community-server-5.7.32-1.el7.x86_64 需要
--> 正在检查事务
---> 软件包 mysql-community-client.x86_64.0.5.7.32-1.el7 将被 安装
--> 正在处理依赖关系 mysql-community-libs(x86-64) >= 5.7.9,它被软件包 mysql-community-client-5.7.32-1.el7.x86_64 需要
---> 软件包 mysql-community-common.x86_64.0.5.7.32-1.el7 将被 安装
--> 正在检查事务
---> 软件包 mysql-community-libs.x86_64.0.5.7.32-1.el7 将被 安装
--> 解决依赖关系完成

依赖关系解决

==============================================================================================
 Package                      架构         版本                 源                       大小
==============================================================================================
正在安装:
 mysql-community-server       x86_64       5.7.32-1.el7         mysql57-community       173 M
为依赖而安装:
 mysql-community-client       x86_64       5.7.32-1.el7         mysql57-community        25 M
 mysql-community-common       x86_64       5.7.32-1.el7         mysql57-community       308 k
 mysql-community-libs         x86_64       5.7.32-1.el7         mysql57-community       2.3 M

事务概要
==============================================================================================
安装  1 软件包 (+3 依赖软件包)

总下载量:201 M
安装大小:875 M

....

已安装:
  mysql-community-server.x86_64 0:5.7.32-1.el7                                                

作为依赖被安装:
  mysql-community-client.x86_64 0:5.7.32-1.el7  
  mysql-community-common.x86_64 0:5.7.32-1.el7 
  mysql-community-libs.x86_64 0:5.7.32-1.el7 

启动MySQL服务

systemctl start mysqld.service # 或 service mysqld start

配置开机启动

systemctl enable mysqld.service

查看运行状态

systemctl status mysqld.service

mysql      2574      1  1 23:49 ?        00:00:00 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid

查看到进程信息

netstat -anpl  | grep mysql

查看端口,可以看出mysql server的进程mysqld所使用的默认端口即3306

$ sudo netstat -anpl | grep tcp
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1412/sshd         
tcp        0     52 192.168.186.103:22      192.168.186.1:54058     ESTABLISHED 2287/sshd: hadoop 
tcp6       0      0 :::3306                 :::*                    LISTEN      2574/mysqld       
tcp6       0      0 192.168.186.103:3888    :::*                    LISTEN      2060/java         
tcp6       0      0 :::22                   :::*                    LISTEN      1412/sshd         
tcp6       0      0 :::37791                :::*                    LISTEN      2060/java         
tcp6       0      0 :::2181                 :::*                    LISTEN      2060/java 

找到临时密码

但是要登陆mysql,我们需要root密码,这个密码是安装时随机生成在MySQL的服务器日志中的

grep "temporary password" /var/log/mysqld.log

2020-02-26T17:05:45.104999Z 1 [Note] A temporary password is generated for root@localhost: bl/!6qaU.wuX

登录MySQL

# 回车后输入在日志中找到的临时root登录密码登录
mysql -u roop -p 

问题: 日志里没有找到临时密码, 或者密码不能登录,可能的原因就是之前安装过MySQL

解决办法:

sudo rm -rf /var/lib/mysql

再重启服务后就可以在日志文件中找到重新生成的临时密码了

systemctl restart mysqld.service

$ cat /var/log/mysqld.log | grep "temporary password"
2020-02-26T17:05:45.104999Z 1 [Note] A temporary password is generated for root@localhost: bl/!6qaU.wuX
2020-08-01T16:07:26.941916Z 1 [Note] A temporary password is generated for root@localhost: x/ttqh=)_6Z/

也可以直接使用明文密码登陆,像这样

mysql -uroop -p"x/ttqh=)_6Z/"

初始密码我们可以这样操作,因为很快我们会将将其改为其他的密码。

但是正常使用时不建议这样 不安全

黑客可以使用history命令查看到你在命令行的输入获取到你的MySQL的root密码

另外MySQL报错在/var下面的log目录下可以查看


如果是虚拟机环境为了方便可以选择配置跳过密码验证

vim /etc/my.cnf

添加以下代码

default-authentication-plugin=mysql_native_password #配置文件本来就有去掉注释即可
symbolic-links=0
skip-grant-tables #跳过密码验证

第一次登陆后系统会很快提示你修改掉默认密码

mysql> show databases;
ERROR 1820 (HY000): You must reset your password using ALTER USER statement before executing this statement.

尝试修改密码(虚拟机可以设置简单密码,如果是购买的服务器则需设置复杂密码并修改默认端口号)

mysql> alter user 'root'@'localhost' identified by 'niit1234';
ERROR 1819 (HY000): Your password does not satisfy the current policy requirements

密码安全策略

但是基于密码策略,所设置的密码必须要包含大小写字母、数字和字符。

修改策略并设置简单密码

将策略要求置为0(LOW),长度要求置为1

mysql> set global validate_password_policy=0;
mysql> set global validate_password_length=1;

关于密码安全策略:

通过show命令查看当前的策略

mysql> show variables like 'validate_password%';

+--------------------------------------+--------+
| Variable_name                        | Value  |
+--------------------------------------+--------+
| validate_password_check_user_name    | OFF    |
| validate_password_dictionary_file    |        |
| validate_password_length             | 8      |
| validate_password_mixed_case_count   | 1      |
| validate_password_number_count       | 1      |
| validate_password_policy             | MEDIUM |
| validate_password_special_char_count | 1      |
+--------------------------------------+--------+
7 rows in set (0.00 sec)

1) validate_password_policy:密码安全策略,默认MEDIUM策略

策略 检查规则
0 or LOW Length
1 or MEDIUM Length; numeric, lowercase/uppercase, and special characters
2 or STRONG Length; numeric, lowercase/uppercase, and special characters; dictionary file

2)validate_password_dictionary_file:密码策略文件,策略为STRONG才需要

3)validate_password_length:密码最少长度

4)validate_password_mixed_case_count:大小写字符长度,至少1个

5)validate_password_number_count :数字至少1个

6)validate_password_special_char_count:特殊字符至少1个

这样以后就可以使用新密码登陆了

开启远程登陆

mysql命令的-h选项可以设置需要远程登陆mysql的服务器的ip地址

如果是本地的话可以使用127.0.0.1或者直接省略

但是远程访问因为安全原因默认是关闭的

mysql> select Host,User from user;
+-----------+---------------+
| Host      | User          |
+-----------+---------------+
| localhost | mysql.session |
| localhost | mysql.sys     |
| localhost | root          |
+-----------+---------------+
3 rows in set (0.00 sec)

这里可以看到host都是localhost,这就是因为这些账号只有本地访问的权限

我们需要将root的访问权限扩大都允许任意方式访问(包括远程访问)

GRANT ALL PRIVILEGES ON *.* TO 'root'@'%'IDENTIFIED BY 'niit1234' WITH GRANT OPTION;

也可以创建一个专门用来支持远程访问的单独用户

GRANT ALL PRIVILEGES ON *.* TO 'remote'@'%' IDENTIFIED BY 'niit1234' WITH GRANT OPTION; 

如果你想允许用户myuser从ip为192.168.1.6的主机连接到mysql服务器,并使用mypassword作为密码

若你想限制能连接到mysql服务器的客户端所在的IP地址为'192.168.1.6'

并使用mypassword作为密码,则命令如下:

GRANT ALL PRIVILEGES ON . TO 'mysqluser'@'192.168.1.6'IDENTIFIED BY 'mypassword' WITH GRANT OPTION;  

使修改生效

现在虽然修改了远程访问权限,但是还没有生效

我们需要刷新权限(或者重启服务),当然直接在mysql执行环境刷新权限是更好的方法

mysql>FLUSH PRIVILEGES;

然后我们可以试试使用navicat远程连接虚拟机中mysql试试

navicat虽然是收费的还是可以下载到破解的版本

最后一点

如果你的服务器开启了防火墙也可能导致远程无法连接

因为mysql默认使用3306端口,我们需要开放这个端口给mysql客户端连接

修改编码

默认安装的设置对中文的支持不好,一般需要修改成utf8,但这里我们修改成utf8mb4编码,因为它是uft8的超集,减少更多乱码的问题,也是推荐使用的编码,比如需要插入emoji字符在数据库中,utf8的显示就会出现问题,而utf8mb4则没有问题。

首先查看当前的编码:

mysql> show variables like '%char%';
+--------------------------------------+----------------------------+
| Variable_name                        | Value                      |
+--------------------------------------+----------------------------+
| character_set_client                 | utf8                       |
| character_set_connection             | utf8                       |
| character_set_database               | latin1                     |
| character_set_filesystem             | binary                     |
| character_set_results                | utf8                       |
| character_set_server                 | latin1                     |
| character_set_system                 | utf8                       |
| character_sets_dir                   | /usr/share/mysql/charsets/ |
| validate_password_special_char_count | 1                          |
+--------------------------------------+----------------------------+
9 rows in set (0.00 sec)

为了修改编码,我们需要修改mysql的核心配置文件如下:

[hadoop@hadoop000 download]$ sudo vim /etc/my.cnf

# For advice on how to change settings please see
# For advice on how to change settings please see
# http://dev.mysql.com/doc/refman/5.7/en/server-configuration-defaults.html

[mysqld]
#
# Remove leading # and set to the amount of RAM for the most important data
# cache in MySQL. Start at 70% of total RAM for dedicated server, else 10%.
# innodb_buffer_pool_size = 128M
#
# Remove leading # to turn on a very important data integrity option: logging
# changes to the binary log between backups.
# log_bin
#
# Remove leading # to set options mainly useful for reporting servers.
# The server defaults are faster for transactions and fast SELECTs.
# Adjust sizes as needed, experiment to find the optimal values.
# join_buffer_size = 128M
# sort_buffer_size = 2M
# read_rnd_buffer_size = 2M
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock

# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0

log-error=/var/log/mysqld.log

# 默认服务器内部操作字符集
character-set-server=utf8mb4
# 默认服务器内部操作字符集校对规则
collation-server=utf8mb4_general_ci
# 默认的存储引擎
default-storage-engine=InnoDB
# 初始化连接时设置以下字符集:
# character_set_client
# character_set_results
# character_set_connection
init_connect='set names utf8mb4'

[client]
default-character-set=utf8mb4

[mysql]
default-character-set=utf8mb4
  • 30行以下为手动添加的配置

验证

mysql> show variables like '%char%';
+--------------------------------------+----------------------------+
| Variable_name                        | Value                      |
+--------------------------------------+----------------------------+
| character_set_client                 | utf8mb4                    |
| character_set_connection             | utf8mb4                    |
| character_set_database               | utf8mb4                    |
| character_set_filesystem             | binary                     |
| character_set_results                | utf8mb4                    |
| character_set_server                 | utf8mb4                    |
| character_set_system                 | utf8                       |
| character_sets_dir                   | /usr/share/mysql/charsets/ |
| validate_password_special_char_count | 1                          |
+--------------------------------------+----------------------------+
9 rows in set (0.00 sec)
  • character_set_client 是指客户端发送过来的语句的编码
  • character_set_database 是指服务器内部使用的编码
  • character_set_connection 是指mysqld收到客户端的语句后,要转换到的编码
  • character_set_results 是指server执行语句后,返回给客户端的数据的编码
  • character_set_system是元数据编码,无需修改
  • character_set_filesystem是文件系统的编码,2进制存储最有效,不能修改

【END】

rpm-bundle的离线安装

下载rpm-bundle包


$ wget http://mirrors.163.com/mysql/Downloads/MySQL-5.7/mysql-5.7.33-1.el7.x86_64.rpm-bundle.tar
$  tar xvf mysql-5.7.33-1.el7.x86_64.rpm-bundle.tar
$  mkdir mysql-jars
$  mv mysql-comm*.rpm mysql-jars/
$  cd mysql-jars/
$ ls
mysql-community-client-5.7.33-1.el7.x86_64.rpm
mysql-community-common-5.7.33-1.el7.x86_64.rpm
mysql-community-libs-5.7.33-1.el7.x86_64.rpm
mysql-community-libs-compat-5.7.33-1.el7.x86_64.rpm
mysql-community-server-5.7.33-1.el7.x86_64.rpm

依次手动安装

sudo rpm -ivh mysql-community-common-5.7.33-1.el7.x86_64.rpm
sudo rpm -ivh mysql-community-libs-5.7.33-1.el7.x86_64.rpm
sudo rpm -ivh mysql-community-libs-compat-5.7.33-1.el7.x86_64.rpm
sudo rpm -ivh mysql-community-client-5.7.33-1.el7.x86_64.rpm
sudo rpm -ivh mysql-community-server-5.7.33-1.el7.x86_64.rpm

Views: 74

04 配置HBase伪分布式环境

如果使用Hadoop3.1.x,建议配合Hbase2使用,如HBase2.2.3。

Hadoop version support matrix

  • √ = Tested to be fully-functional
  • ! = Known to not be fully-functional, or there are CVEs so we drop the support in newer minor releases
  • x = Not tested, may/may-not function
HBase-1.4.x HBase-1.6.x HBase-1.7.x HBase-2.2.x HBase-2.3.x
Hadoop-2.7.0 x x x x x
Hadoop-2.7.1+ √ x x x x
Hadoop-2.8.[0-2] x x x x x
Hadoop-2.8.[3-4] ! x x x x
Hadoop-2.8.5+ ! √ x √ x
Hadoop-2.9.[0-1] x x x x x
Hadoop-2.9.2+ ! √ x √ x
Hadoop-2.10.x ! √ √ ! √
Hadoop-3.1.0 x x x x x
Hadoop-3.1.1+ x x x √ √
Hadoop-3.2.x x x x √ √

1、解压

[hadoop@hadoop100 hbase-2.2.3]$ tar -zxvf hbase-2.2.3-bin.tar.gz -C /opt/pkg/

2、环境变量

[hadoop@hadoop100 hbase-2.2.3]$ vi conf/hbase-env.sh
[hadoop@hadoop100 hbase-2.2.3]$ sudo vim /etc/profile.d/env.sh 

# JAVA_HOME
export JAVA_HOME=/opt/pkg/java
export PATH=$JAVA_HOME/bin:$PATH

# HADOOP_HOME
export HADOOP_HOME=/opt/pkg/hadoop-3.1.4
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

# HBASE_HOME
export HBASE_HOME=/opt/pkg/hbase-2.2.3
export PATH=$HBASE_HOME/bin:$PATH

最后使用source命令使配置生效

[hadoop@hadoop100 hbase-2.2.3]$ source /etc/profile.d/env.sh

3、配置

拷贝hadoop的hdfs-site.xml和core-site.xml软链接到hbase的conf目录下面


# core-site.xml
$ ln -s /opt/pkg/hadoop-2.7.3/etc/hadoop/hdfs-site.xml /opt/pkg/hbase-2.2.3/conf/hdfs-site.xml
# hdfs-site.xml
$ ln -s /opt/pkg/hadoop-2.7.3/etc/hadoop/core-site.xml /opt/pkg/hbase-2.2.3/conf/core-site.xml

修改hbase的conf/hbase-env.sh

# The java implementation to use.  Java 1.7+ required.
# export JAVA_HOME=/usr/java/jdk1.6.0/
export JAVA_HOME=/opt/pkg/java

# Configure PermSize. Only needed in JDK7. You can safely remove it for JDK8+
# export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS -XX:PermSize=128m -XX:MaxPermSize=128m"
# export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -XX:PermSize=128m -XX:MaxPermSize=128m"

# Tell HBase whether it should manage it's own instance of Zookeeper or not.
# export HBASE_MANAGES_ZK=true
  • 首先,需要修改JAVA_HOME为真实JDK路径

  • Configure PermSize.下面的2条export语句只是针对JDK7的优化,如果不用JDK7则可以删除或者注释掉

  • 默认情况,由HBase自行管理内置在其内的Zookeeper服务

    • 如果想要使用外部Zookeeper集群来进行管理,需要设置
    export HBASE_MANAGES_ZK=false

修改hbase的核心配置文件conf/hbase-site.xml

<configuration>
    <!-- Hbase在HDFS上的根目录 -->
    <property>
        <name>hbase.rootdir</name>
        <value>hdfs://hadoop100:8020/hbase</value>
    </property>
    <!-- HBase在ZooKeeper上的快照数据存储位置 -->
    <property>
        <name>hbase.zookeeper.property.dataDir</name>
        <value>/opt/pkg/hadoop/data/tmp/hbase-zkdata</value>
    </property>
    <!-- 是否以集群的方式运行 -->
    <property>
        <name>hbase.cluster.distributed</name>
        <value>true</value>
    </property>
    <!-- Zookeeper集群的所有主机,如有多个则逗号分隔 -->
    <property>
        <name>hbase.zookeeper.quorum</name>
        <value>hadoop100</value>
    </property>
</configuration>
  • hbase.rootdir是hbase存储在HDFS上的数据的根目录,里面存储着所有region的数据

  • 无论是伪分布式还是全分布式,hbase.cluster.distributed都需要设置true

    • 如果设成false,则hbase和zookeeper会在一个JVM进程里运行
  • hbase.zookeeper.quorum的主机名默认是localhost,应改为zookeeper集群中所有机器的主机名,多个主机名之间使用逗号分隔

  • zookeeper的端口默认都是2181,如果不是可以添加

问题:

java.lang.IllegalStateException: The procedure WAL relies on the ability to hsync for proper operation during component failures, but the underlying filesystem does not support doing so. Please check the config value of 'hbase.procedure.store.wal.use.hsync' to set the desired level of robustness and ensure the config value of 'hbase.wal.dir' points to a FileSystem mount that can provide it.

解决:

  • 在hbase-site.xml增加配置
<property>
    <name>hbase.unsafe.stream.capability.enforce</name>
    <value>false</value>
</property>

4、测试

首先启动hadoop的dfs相关进程

[hadoop@hadoop100 lib]$ start-dfs.sh 

等半分钟后启动Hbase

[hadoop@hadoop100 lib]$ start-hbase.sh 
hadoop100: starting zookeeper, logging to /opt/pkg/hbase-2.2.3/logs/hbase-hadoop-zookeeper-hadoop100.out
starting master, logging to /opt/pkg/hbase-2.2.3/logs/hbase-hadoop-master-hadoop100.out
starting regionserver, logging to /opt/pkg/hbase-2.2.3/logs/hbase-hadoop-1-regionserver-hadoop100.out

使用hbase shell测试


[hadoop@hadoop100 lib]$ hbase shell
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 2.2.3, r67592f3d062743907f8c5ae00dbbe1ae4f69e5af, Tue Oct 25 18:10:20 CDT 2020

hbase(main):001:0> status
1 active master, 0 backup masters, 1 servers, 0 dead, 2.0000 average load

hbase(main):002:0> list
TABLE                                                                                                   
0 row(s) in 0.0490 seconds

=> []

hbase(main):004:0> create 'wc','cf'
0 row(s) in 4.6740 seconds

=> Hbase::Table - wc
hbase(main):005:0> put 'wc','hello', 'cf:word', 'word'
0 row(s) in 0.2030 seconds

hbase(main):006:0> scan 'wc'
ROW                         COLUMN+CELL                                                                 
 hello                      column=cf:word, timestamp=1610209834376, value=word                         
1 row(s) in 0.0590 seconds

hbase(main):007:0> 

界面

hbase的Master管理界面地址: http://hadoop100:16010

image-20210110012902737

hbase的RegionServer管理界面地址: http://hadoop100:16301

image-20210110013140999

常见问题

注意:如果启动HBase Shell时遇到警告:

[hadoop@hadoop100 hbase-2.2.3]$ hbase shell
SLF4J: Class path contains multiple SLF4J bindings.

原因是hadoop和hbase的jar包冲突了,解决办法,将hbase/lib下面的相同slf4j-log4j12的jar包改名即可

[hadoop@hadoop100 lib]$ mv slf4j-log4j12-1.7.5.jar slf4j-log4j12-1.7.5.jar.backup

Views: 48

03- Zookeeper安装和配置

1、下载和安装

下载,解压,配置环境变量就不多说了,和其他框架都大致一样,版本选择3.14

2、单机单服务

zoo.cfg

tickTime=2000
initLimit=10
syncLimit=5
dataDir=/opt/tmp/zk
dataLogDir=/opt/tmp/zk-log

3、单机多服务

img

三个配置文件zoo-1.cfg、zoo-2.cfg、zoo-3.cfg

tickTime=2000
initLimit=10
syncLimit=5
dataDir=/opt/tmp/zk-1(zk-2|zk-3)
dataLogDir=/opt/tmp/zk-log-1(zk-log-2|zk-log-3)
clientPort=2181(2182|2183)

4lw.commands.whitelist=*
server.1=localhost:2891:3891
server.2=localhost:2892:3892
server.3=localhost:2893:3893

在三个dataDir里面分别建立一个myid文件, 里面分别保存数字1, 2, 3,对应配置中的server。

4、zk服务相关命令

zkServer.sh start|status|stop zoo-1|2|3.cfg

新版本的kafka里面自带zookeeper,你可以在bin里面找到自带的zookeeper相关命令

zookeeper的提供了模板配置文件zoo-sample.cfg,可以拷贝修改成为自己的配置

Views: 53

02 – Hadoop3.1.4的单机安装

1、准备工作

(1)主机名映射S

# vi /etc/hosts vi /etc/hosts

#127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
#::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.186.100 hadoop100

注意:宿主的WIndows系统也要一样配置hadoop100并且映射到虚拟机的ip上

(2)SSH免密登录

Centos7默认安装了sshd服务,可使用ssh协议远程开启对其他主机shell的服务。

如果没有可以使用yum安装

sudo yum -y install openssl-devel

由于Hadoop集群的机器之间ssh通信默认需要输入密码,在集群运行时我们不可能为每一次通信都手动输入密码,因此需要配置机器之间的ssh的免密登录。

即使是单机伪分布式的Hadoop环境也不例外,一样需要配置本地对本地ssh连接的免密。

hadoop@hadoop100 ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): 
Created directory '/home/hadoop/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:yQYChs4eniVLeICaI2bCB9HopbXUBE9v0lpLjBACUnM hadoop@hadoop100
The key's randomart image is:
+---[RSA 2048]----+
|==O+Eo           |
|=+.O+.=          |
|Bo* o+.B         |
|B%.+ .*o..       |
|OoB  . .S        |
| =     .         |
|                 |
|                 |
|                 |
+----[SHA256]-----+
[hadoop@hadoop100 ~]$ cd ~/.ssh/
[hadoop@hadoop100 .ssh]$ cat id_rsa.pub >> authorized_keys
[hadoop@hadoop100 .ssh]$ chmod 600 authorized_keys 
[hadoop@hadoop100 .ssh]$ ssh hadoop@hadoop100
The authenticity of host 'hadoop100 (192.168.186.100)' can't be established.
ECDSA key fingerprint is SHA256:aGLhdt3bIuqtPgrFWnhgrfTKUbDh4CWVTfIgr5E5oV0.
ECDSA key fingerprint is MD5:b8:bd:b3:65:fe:77:2c:06:2d:ec:58:3a:97:51:dd:ca.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'hadoop100,192.168.186.100' (ECDSA) to the list of known hosts.
Last login: Sat Jan  9 10:16:53 2021 from 192.168.186.1
[hadoop@hadoop100 ~]$ exit
登出
Connection to hadoop100 closed.
[hadoop@hadoop100 .ssh]$

大致流程:

  1. ssh-keygen命令生成RSA加密的密钥对(公钥和私钥)
  2. 将公钥添加到~/.ssh目录下的authorized_keys文件中
    1. 如机器A需要SSH连接到机器B,就需要将机器A生成的公钥发送到机器B的authorized_keys文件中进行认证
    2. 如果需要和自己通信,就把自己生成的公钥放在自己的authorized_keys文件中即可
  3. 使用ssh命令连接本地终端,如果不需要输入密码则本地的免密配置成功

(3)配置时间同步

集群中的通信和文件传输一般是以系统时间作为约定条件的。所以当集群中机器之间系统如果不一致可能导致各种问题发生,比如访问时间过长,甚至失败。所以配置机器之间的时间同步非常重要。

另外,单机伪分布式可以无需配置时间同步服务器,只需要定时使用nptdate命令校准即可。

以下操作必须切换成root用户

安装ntp

[hadoop@hadoop100 .ssh]$ su root
密码:
[root@hadoop100 .ssh]# rpm -qa | grep ntp
[root@hadoop100 .ssh]# yum install -y ntp
  1. rpm -qa查询是否已经安装了ntp
  2. 如果没有则安装ntp

修改/etc/sysconfig/ntpd,增加如下内容(让硬件时间和系统时间一起同步)

SYNC_HWCLOCK=yes

(4)定时校正时间

使用ntpdate命令手动的校正时间。

[root@hadoop100 .ssh]#  ntpdate cn.pool.ntp.org
 9 Jan 11:49:13 ntpdate[11274]: adjust time server 185.209.85.222 offset -0.016417 sec
[root@hadoop100 .ssh]# date
2021年 01月 09日 星期六 11:49:32 CST

观察时间是否与你所在的时区一致,如果不一致请查看【常见问题】寻找解决办法。

接下来使用crontab定时任务来每10分钟校准一次

# 编辑新的定时任务
[root@hadoop100 .ssh]# crontab -e
*/10 * * * * /usr/sbin/ntpdate cn.pool.ntp.org

# 查看root账户下所有定时任务
[root@hadoop100 .ssh]# crontab -l
*/10 * * * * /usr/sbin/ntpdate cn.pool.ntp.org

为了测试,先修改时间为错误时间

[root@hadoop100 .ssh]# date -s '2020-1-1 01:01'
2020年 01月 01日 星期三 01:01:00 CST 

10分钟后再次查看

[root@hadoop100 .ssh]# date
2021年 01月 09日 星期六 14:12:12 CST

说明自动校准已经设置

默认crontab定时任务就是开机自动启动的

[root@hadoop100 .ssh]# systemctl list-unit-files | grep crond
crond.service                                 enabled 

(5)统一目录结构

接下来切换用户为hadoop账户

[hadoop@hadoop100 opt]$ sudo mkdir /opt/download
[hadoop@hadoop100 opt]$ sudo mkdir /opt/data
[hadoop@hadoop100 opt]$ sudo mkdir /opt/bin
[hadoop@hadoop100 opt]$ sudo mkdir /opt/tmp
[hadoop@hadoop100 opt]$ sudo mkdir /opt/pkg

# 更改opt下目录的用户及其所在用户组为hadoop
[hadoop@hadoop100 opt]$ sudo chown hadoop:hadoop /opt/*
[hadoop@hadoop100 opt]$ ls
bin  data  download  pkg  tmp
[hadoop@hadoop100 opt]$ ll
总用量 0
drwxr-xr-x. 2 hadoop hadoop 6 1月   9 14:29 bin
drwxr-xr-x. 2 hadoop hadoop 6 1月   9 14:29 data
drwxr-xr-x. 2 hadoop hadoop 6 1月   9 14:29 download
drwxr-xr-x. 2 hadoop hadoop 6 1月   9 14:37 pkg
drwxr-xr-x. 2 hadoop hadoop 6 1月   9 14:36 tmp

目录规划如下:

/opt/ 
    ├── bin         # shell脚本
    ├── data        # 程序需要使用的数据
    ├── download    # 下载的软件安装包
    ├── pkg         # 解压方式安装的软件
    └── tmp         # 存放程序生成的临时文件

2、安装jdk

检查是否已经安装过JDK

[hadoop@hadoop100 download]$ rpm -qa | grep java
# 或者
[hadoop@hadoop100 download]$ yum list installed | grep java

如果没有jdk或者jdk版本低于1.8,则重新安装jdk1.8

下载安装包到/opt/download/然后解压到/opt/pkg

[hadoop@hadoop100 download]$ tar -zxvf jdk-8u261-linux-x64.tar.gz 
[hadoop@hadoop100 download]$ mv jdk1.8.0_261 /opt/pkg/java

配置java环境变量

确认jdk的解压路径

[hadoop@hadoop100 java]$ pwd
/opt/pkg/java

编辑/etc/profile.d/env.sh配置文件(没有则创建)

[hadoop@hadoop100 java]$ sudo vim /etc/profile.d/env.sh

在后面添加新的环境变量配置

# JAVA_HOME
export JAVA_HOME=/opt/pkg/java
export PATH=$JAVA_HOME/bin:$PATH

使新的环境变量立刻生效

[hadoop@hadoop100 java]$ source /etc/profile.d/env.sh

验证环境变量

[hadoop@hadoop100 java]$ java -version
[hadoop@hadoop100 java]$ java
[hadoop@hadoop100 java]$ javac

3、安装Hadoop

  1. 解压

    [hadoop@hadoop100 hadoop]$ tar -zxvf hadoop.tar.gz -C /opt/pkg/
  2. 编辑/etc/profile.d/env.sh配置文件,添加环境变量

    # JAVA_HOME
    export JAVA_HOME=/opt/pkg/java
    export PATH=$JAVA_HOME/bin:$PATH
    # HADOOP_HOME
    export HADOOP_HOME=/opt/pkg/hadoop
    export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
  3. 使新的环境变量立刻生效

    [hadoop@hadoop100 opt]$ source /etc/profile.d/env.sh
  4. 验证

    [hadoop@hadoop100 opt]$ hadoop version
  5. 修改相关命令执行环境

    hadoop/etc/hadoop/hadoop-env.sh - hadoop命令执行环境

      # The java implementation to use.
      export JAVA_HOME=/opt/pkg/java
    • 修改JAVA_HOME为真实JDK路径即可
  6. hadoop/etc/hadoop/yarn.env.sh - yarn命令执行环境

      # export JAVA_HOME=/home/y/libexec/jdk1.6.0/
      export JAVA_HOME=/opt/pkg/java
    • 添加JAVA_HOME为真实JDK路径即可
  7. hadoop/etc/hadoop/mapred.env.sh - map reducer命令执行环境

      # export JAVA_HOME=/home/y/libexec/jdk1.6.0/
      export JAVA_HOME=/opt/pkg/java
    • 添加JAVA_HOME为真实JDK路径即可
  8. 修改配置-实现伪分布式环境,来到 hadoop/etc/hadoop/,修改以下配置文件

    hadoop/etc/hadoop/core-site.xml - 核心配置文件

      <configuration>
          <!-- 指定NameNode的地址和端口. -->
          <property>
              <name>fs.defaultFS</name>
              <value>hdfs://hadoop100:8020</value>
          </property>
          <!-- 指定HDFS系统运行时产生的文件的存储目录. -->
          <property>
              <name>hadoop.tmp.dir</name>
              <value>/opt/pkg/hadoop/data/tmp</value>
          </property>
          <!--  缓冲区大小,实际工作中根据服务器性能动态调整;默认值4096 -->
       <property>
           <name>io.file.buffer.size</name>
           <value>4096</value>
       </property>
       <!--  开启hdfs的垃圾桶机制,删除掉的数据可以从垃圾桶中回收,单位分钟;默认值0 -->
       <property>
           <name>fs.trash.interval</name>
           <value>10080</value>
       </property>
    </configuration>
    1. 主机名修改成本机的主机名

    2. hadoop.tmp.dir十分重要,保存这个hadoop集群中namenode和datanode的所有数据

  9. hadoop/etc/hadoop/hdfs-site.xml - HDFS相关配置

      <configuration>
          <!-- 设置HDFS中的数据副本数. -->
          <property>
             <name>dfs.replication</name>
             <value>1</value>
          </property>
           <!-- 设置Hadoop的Secondary NameNode的主机配置 -->
          <property>
             <name>dfs.namenode.secondary.http-address</name>
             <value>hadoop100:9868</value>
          </property>
          <property>
              <name>dfs.namenode.http-address</name>
              <value>hadoop100:9870</value>
          </property>
          <!-- 是否检查操作HDFS文件系统的用户权限. -->
          <property>
          <name>dfs.permissions</name>
          <value>false</value>
       </property>
    </configuration>
    • dfs.replication默认是3,为了节省虚拟机资源,这里设置为1

    • 全分布式情况下,SecondaryNameNode和NameNode 应分开部署

    • dfs.namenode.secondary.http-address默认就是本地,如果是伪分布式可以不用配置

  10. hadoop/etc/hadoop/mapred-site.xml - mapreduce 相关配置

    <configuration>
        <!-- 指定MapReduce程序由Yarn进行调度. -->
        <property>
            <name>mapreduce.framework.name</name>
            <value>yarn</value>
        </property>
            <!-- Mapreduce的Job历史记录服务器主机端口设置. -->
        <property>
            <name>mapreduce.jobhistory.address</name>
            <value>hadop100:10020</value>
        </property>
            <!-- Mapreduce的Job历史记录的Webapp端地址. -->
        <property>
            <name>mapreduce.jobhistory.webapp.address</name>
            <value>hadoop100:19888</value>
        </property>
    
        <property>
            <name>yarn.app.mapreduce.am.env</name>
            <value>HADOOP_MAPRED_HOME=/opt/pkg/hadoop</value>
        </property>
        <property>
            <name>mapreduce.map.env</name>
            <value>HADOOP_MAPRED_HOME=/opt/pkg/hadoop</value>
        </property>
        <property>
            <name>mapreduce.reduce.env</name>
            <value>HADOOP_MAPRED_HOME=/opt/pkg/hadoop</value>
        </property>
    
    </configuration>

    ​

    • mapreduce.jobhistory相关配置是可选配置,用于查看MR任务的历史日志

      • 这里主机名千万不要弄错,不然任务执行会失败,且不容易找原因
        • 需要手动启动MapReduceJobHistory后台服务才能在Yarn的页面打开历史日志
  11. 配置 yarn-site.xml

      <configuration>
          <!-- 设置Yarn的ResourceManager节点主机名. -->
        <property>
           <name>yarn.resourcemanager.hostname</name>
           <value>hadoop100</value>
        </property>
           <!-- 设置Mapper端将数据发送到Reducer端的方式. -->
        <property>
           <name>yarn.nodemanager.aux-services</name>
           <value>mapreduce_shuffle</value>
        </property>
          <!-- 是否开启日志手机功能. -->
        <property>
           <name>yarn.log-aggregation-enable</name>
           <value>true</value>
        </property>
          <!-- 日志保留时间(7天). -->
        <property>
           <name>yarn.log-aggregation.retain-seconds</name>
           <value>604800</value>
        </property>
          <!-- 如果vmem、pmem资源不够,会报错,此处将资源监察置为false -->
       <property>
           <name>yarn.nodemanager.vmem-check-enabled</name>
           <value>false</value>
       </property>
       <property>
           <name>yarn.nodemanager.pmem-check-enabled</name>
           <value>false</value>
       </property>
    </configuration>
  12. workers DataNode 节点配置

      vi workers
      [hadoop@hadoop100 hadoop]$ vi workers
      hadoop100
    • 如果是伪分布式可以不进行修改,默认是localhost, 也可以改成本机的主机名
  • 全分布式配置则需要每行输入一个DataNode主机名

    • 注意不要有空格和空行,因为其他脚本会获取相关主机名信息

4、格式化名称节点

格式化:HDFS(NameNode)

hadoop@hadoop100 hadoop]$ hdfs namenode -format

21/01/09 19:27:21 INFO namenode.NameNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = hadoop100/192.168.186.100
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 2.7.3
************************************************************/
21/01/09 19:27:21 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
21/01/09 19:27:21 INFO namenode.NameNode: createNameNode [-format]
Formatting using clusterid: CID-08318e9e-e202-48f3-bcb1-548ca50310c9
21/01/09 19:27:22 INFO util.GSet: Computing capacity for map BlocksMap
21/01/09 19:27:22 INFO util.GSet: VM type       = 64-bit
21/01/09 19:27:22 INFO util.GSet: 2.0% max memory 966.7 MB = 19.3 MB
21/01/09 19:27:22 INFO util.GSet: capacity      = 2^21 = 2097152 entries
21/01/09 19:27:22 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
21/01/09 19:27:22 INFO blockmanagement.BlockManager: defaultReplication         = 1
21/01/09 19:27:22 INFO blockmanagement.BlockManager: maxReplication             = 512
21/01/09 19:27:22 INFO blockmanagement.BlockManager: minReplication             = 1
21/01/09 19:27:22 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2
21/01/09 19:27:22 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
21/01/09 19:27:22 INFO blockmanagement.BlockManager: encryptDataTransfer        = false
21/01/09 19:27:22 INFO blockmanagement.BlockManager: maxNumBlocksToLog          = 1000
21/01/09 19:27:22 INFO namenode.FSNamesystem: fsOwner             = hadoop (auth:SIMPLE)
21/01/09 19:27:22 INFO namenode.FSNamesystem: supergroup          = supergroup
21/01/09 19:27:22 INFO namenode.FSNamesystem: isPermissionEnabled = false
21/01/09 19:27:22 INFO namenode.FSNamesystem: HA Enabled: false
21/01/09 19:27:22 INFO namenode.FSNamesystem: Append Enabled: true
21/01/09 19:27:23 INFO common.Storage: Storage directory /opt/pkg/hadoop/data/tmp/dfs/name has been successfully formatted.
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at hadoop100/192.168.186.100
************************************************************/
  • 注意格式化后会在hdfs-site.xml中指定的hadoop.tmp.dir目录中生成相关数据

  • 其中NameNode和DataNode的数据文件夹中应保存着一致的ClusterID(CID)

    [hadoop@hadoop100 hadoop]$ cat /opt/pkg/hadoop/data/tmp/dfs/name/current/VERSION
    #Sat Jan 09 19:27:23 CST 2021
    namespaceID=637773384
    clusterID=CID-08318e9e-e202-48f3-bcb1-548ca50310c9
    cTime=0
    storageType=NAME_NODE
    blockpoolID=BP-1926974917-192.168.186.100-1610191643341
    layoutVersion=-63
    
    [hadoop@hadoop100 hadoop]$ cat /opt/pkg/hadoop/data/tmp/dfs/data/current/VERSION 
    #Sat Jan 09 19:33:49 CST 2021
    storageID=DS-6abf02d0-274c-4b7a-9d1d-05ed7d73636a
    clusterID=CID-08318e9e-e202-48f3-bcb1-548ca50310c9
    cTime=0
    datanodeUuid=44ff2304-01b1-4d8a-8a42-a2ad6e62ebba
    storageType=DATA_NODE
    layoutVersion=-56
    
    • 如果多次格式化就会导致NamdeNode新生成的CID和DataNode不一致
    • 解决办法,停止集群,将DN的CID修改成和NN的CID一致,再启动集群

5、运行和测试

启动Hadoop环境,刚启动Hadoop的HDFS系统后会有几秒的安全模式,安全模式期间无法进行任何数据处理,这也是为什么不建议使用start-all.sh脚本一次性启动DFS进程和Yarn进程,而是先启动dfs后过30秒左右再启动Yarn相关进程。

启动DFS进程:

[hadoop@hadoop100 hadoop]$ start-dfs.sh
Starting namenodes on [hadoop100]
hadoop100: starting namenode, logging to /opt/pkg/hadoop/logs/hadoop-hadoop-namenode-hadoop100.out
hadoop100: starting datanode, logging to /opt/pkg/hadoop/logs/hadoop-hadoop-datanode-hadoop100.out
Starting secondary namenodes [hadoop100]
hadoop100: starting secondarynamenode, logging to /opt/pkg/hadoop/logs/hadoop-hadoop-secondarynamenode-hadoop100.out

启动YARN进程:

[hadoop@hadoop100 hadoop]$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /opt/pkg/hadoop/logs/yarn-hadoop-resourcemanager-hadoop100.out
hadoop100: starting nodemanager, logging to /opt/pkg/hadoop/logs/yarn-hadoop-nodemanager-hadoop100.out

启动MapReduceJobHistory后台服务 - 用于查看MR执行的历史日志

[hadoop@hadoop100 mapreduce]$ mr-jobhistory-daemon.sh start historyserver

停止集群(可以做成脚本)

stop-dfs.sh
stop-yarn.sh 
# 已过时 mr-jobhistory-daemon.sh stop historyserver
mapred --daemon stop historyserver

单个进程逐个启动

# 在主节点上使用以下命令启动 HDFS NameNode: 
# 已过时 hadoop-daemon.sh start namenode 
hdfs --daemon start namenode

# 在主节点上使用以下命令启动 HDFS SecondaryNamenode: 
# 已过时 hadoop-daemon.sh start secondarynamenode 
hdfs --daemon start secondarynamenode

# 在每个从节点上使用以下命令启动 HDFS DataNode: 
# 已过时 hadoop-daemon.sh start datanode
hdfs --daemon start datanode

# 在主节点上使用以下命令启动 YARN ResourceManager: 
# 已过时 yarn-daemon.sh start resourcemanager 
yarn --daemon start resourcemanager

# 在每个从节点上使用以下命令启动 YARN nodemanager: 
# 已过时 yarn-daemon.sh start nodemanager 
yarn --daemon start nodemanager

以上脚本位于$HADOOP_HOME/sbin/目录下。如果想要停止某个节点上某个角色,只需要把命令中的start 改为stop 即可。

Web界面进行验证

HDFS:http://hadoop100:9870

image-20210109193629973

Yarn:http://hadoop100:8088

image-20210109193653338

执行jps命令,看看是否会有如下进程:

[hadoop@hadoop100 hadoop]$ jps
14608 NodeManager
14361 SecondaryNameNode
14203 DataNode
14510 ResourceManager
14079 NameNode

14910 Jps

使用官方自带的示例程序测试

[hadoop@hadoop100 mapreduce]$ cd /opt/pkg/hadoop/share/hadoop/mapreduce
[hadoop@hadoop100 mapreduce]$ hadoop jar hadoop-mapreduce-examples-2.7.3.jar wordcount
Usage: wordcount <in> [<in>...] <out>

准备输入文件,并上传到HDFS系统

[hadoop@hadoop100 input]$ cat /opt/data/mapred/input/wc.txt
hadoop hadoop hadoop
hi hi hi hello hadoop
hello world hadoop

[hadoop@hadoop100 input]$ hadoop fs -mkdir -p /input/wc
[hadoop@hadoop100 input]$ hadoop fs -put wc.txt /input/wc/
Found 1 items
-rw-r--r--   1 hadoop supergroup         62 2021-01-09 20:15 /input/wc/wc.txt

[hadoop@hadoop100 input]$ hadoop fs -cat /input/wc/wc.txt
hadoop hadoop hadoop
hi hi hi hello hadoop
hello world hadoop

运行示例wordcount程序,并将结果输出到/output/wc之中

[hadoop@hadoop100 mapreduce]$ cd /opt/pkg/hadoop/share/hadoop/mapreduce
[hadoop@hadoop100 mapreduce]$ hadoop jar hadoop-mapreduce-examples-2.7.3.jar wordcount /input/wc/ /output/wc/
21/01/09 20:23:27 INFO client.RMProxy: Connecting to ResourceManager at hadoop100/192.168.186.100:8032
21/01/09 20:23:28 INFO input.FileInputFormat: Total input paths to process : 1
21/01/09 20:23:28 INFO mapreduce.JobSubmitter: number of splits:1
21/01/09 20:23:29 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1610194940581_0001
21/01/09 20:23:29 INFO impl.YarnClientImpl: Submitted application application_1610194940581_0001
21/01/09 20:23:29 INFO mapreduce.Job: The url to track the job: http://hadoop100:8088/proxy/application_1610194940581_0001/
21/01/09 20:23:29 INFO mapreduce.Job: Running job: job_1610194940581_0001
21/01/09 20:23:43 INFO mapreduce.Job: Job job_1610194940581_0001 running in uber mode : false
21/01/09 20:23:43 INFO mapreduce.Job:  map 0% reduce 0%
21/01/09 20:23:52 INFO mapreduce.Job:  map 100% reduce 0%
21/01/09 20:24:00 INFO mapreduce.Job:  map 100% reduce 100%
21/01/09 20:24:01 INFO mapreduce.Job: Job job_1610194940581_0001 completed successfully
21/01/09 20:24:02 INFO mapreduce.Job: Counters: 49
    File System Counters
        FILE: Number of bytes read=52
        FILE: Number of bytes written=237407
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=164
        HDFS: Number of bytes written=30
        HDFS: Number of read operations=6
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=2
    Job Counters 
        Launched map tasks=1
        Launched reduce tasks=1
        Data-local map tasks=1
        Total time spent by all maps in occupied slots (ms)=7041
        Total time spent by all reduces in occupied slots (ms)=5566
        Total time spent by all map tasks (ms)=7041
        Total time spent by all reduce tasks (ms)=5566
        Total vcore-milliseconds taken by all map tasks=7041
        Total vcore-milliseconds taken by all reduce tasks=5566
        Total megabyte-milliseconds taken by all map tasks=7209984
        Total megabyte-milliseconds taken by all reduce tasks=5699584
    Map-Reduce Framework
        Map input records=3
        Map output records=11
        Map output bytes=106
        Map output materialized bytes=52
        Input split bytes=102
        Combine input records=11
        Combine output records=4
        Reduce input groups=4
        Reduce shuffle bytes=52
        Reduce input records=4
        Reduce output records=4
        Spilled Records=8
        Shuffled Maps =1
        Failed Shuffles=0
        Merged Map outputs=1
        GC time elapsed (ms)=146
        CPU time spent (ms)=2570
        Physical memory (bytes) snapshot=339308544
        Virtual memory (bytes) snapshot=4163043328
        Total committed heap usage (bytes)=219676672
    Shuffle Errors
        BAD_ID=0
        CONNECTION=0
        IO_ERROR=0
        WRONG_LENGTH=0
        WRONG_MAP=0
        WRONG_REDUCE=0
    File Input Format Counters 
        Bytes Read=62
    File Output Format Counters 
        Bytes Written=30
  • 注意,输入是文件夹,可以指定多个
  • 输出是一个必须不能存在的文件夹路径
  • 计算结果会写入到output指定的文件夹中

查看保存在HDFS上的结果

[hadoop@hadoop100 mapreduce]$ hadoop fs -ls /output/wc/
Found 2 items
-rw-r--r--   1 hadoop supergroup          0 2021-01-09 20:23 /output/wc/_SUCCESS
-rw-r--r--   1 hadoop supergroup         30 2021-01-09 20:23 /output/wc/part-r-00000
[hadoop@hadoop100 mapreduce]$ hadoop fs -cat /output/wc/part-r-00000
hadoop  5
hello   2
hi  3
world   1

在MR任务执行时,可以通过Yarn的界面查看进度

image-20210109203547507

执行完毕以后点击TrackingUI下的History可以查看历史日志记录

如果跳转页面报404说明没有启动JobHistoryServer服务

[hadoop@hadoop100 mapreduce]$ mr-jobhistory-daemon.sh start historyserver

image-20210109224932545

也可以在HDFS的Web界面上查看结果

image-20210109213522691

Utilities -> HDFS browser -> /output/wc/ -> click part-r-00000 -> download part-r-00000

[hadoop@hadoop100 hadoop]$ stop-all.sh

This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh
Stopping namenodes on [hadoop100]
hadoop100: stopping namenode
hadoop100: stopping datanode
Stopping secondary namenodes [hadoop100]
hadoop100: stopping secondarynamenode
stopping yarn daemons
stopping resourcemanager
hadoop100: stopping nodemanager
no proxyserver to stop

6、常见问题:

(1)时间校准后仍然偏差几个小时

使用nptdate 进行时间校准后发现校准后的时间仍然偏移很多,这是因为时区的设置问题(如果是纽约时间, 则相差13个小时),修改时区的方法就是删除原来的软链。软链的位置:

ll /etc/localtime
/etc/localtime -> ../usr/share/zoneinfo/America/New_York

rm /etc/localtime

删除后时间就变成了UTC时间 - 格林威治时间,与中国时间相差8个小时

接下来只需要重新生成一个指向中国时区的软链,就可以正常显示中国时区的时间了

$ sudo ln -s /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

这样时区就没问题了

(2)没有DataNode

如果多次格式化会导致NamdeNode新生成的CID和DataNode不一致

  • 解决办法,

    • 停止集群,将DN的CID修改成和NN的CID一致,格式化NN,再启动集群
    • 建议
    • 停止集群,删除hadoop.tmp.dir下的所有内容,格式化NN,再启动集群
    • 不建议,会丢失数据,如果不怕丢失数据可以这样做

(3)MapReducer任务失败

有很多原因,主要是分析日志的信息

一般的原因是:

  • 缺少服务进程

    • 查找对应服务的日志
  • 输出目录已经存在

    • 删除已经存在的目录或者改为不存在的目录
  • Host无法解析

    • 检查/etc/hosts的映射的主机名和ip是否准确
    • 检查所有配置文件中的主机名和端口号是否正确
  • 其他的错误

    • 可以通过MR的历史日志进行定位

    • 或者通过MR任务的Yarn日志查看任务执行细节

    yarn logs -applicationId <applicationId>

Views: 117