09 – SQOOP安装配置

1、下载和安装

  1. 下载 http://archive.apache.org/dist/sqoop/1.4.7/
wget http://archive.apache.org/dist/sqoop/1.4.7/sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz
  1. 上传sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz到/tools目录下

  2. 解压安装, 改名

tar -zxvf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz -C /opt/pkg/
cd /opt/pkg/
mv sqoop-1.4.7.bin__hadoop-2.6.0/ sqoop-1.4.7
  1. 配置环境变量
#sqoop
export SQOOP_HOME=/training/sqoop-1.4.7.bin__hadoop-2.6.0
export PATH=$PATH:$SQOOP_HOME/bin

2、配置

  1. 配置sqoop的环境和配置

​ 把Sqoop可能使用的环境变量都配置上

​ 修改sqoop-env.sh:

$ mv conf/sqoop-env-template.sh conf/sqoop-env.sh
$ vi conf/sqoop-env.sh 

# Set Hadoop-specific environment variables here.

#Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME=/opt/pkg/hadoop

#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/opt/pkg/hadoop

#set the path to where bin/hbase is available
export HBASE_HOME=/opt/pkg/hbase

#Set the path to where bin/hive is available
export HIVE_HOME=/opt/pkg/hive

#Set the path for where zookeper config dir is
export ZOOCFGDIR=/opt/pkg/zookeeper/conf
  1. 修改sqoop-site.xml

​ 具体配置如下文件所示:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

  <property>
    <name>sqoop.metastore.client.enable.autoconnect</name>
    <value>true</value>
    <description>If true, Sqoop will connect to a local metastore
      for job management when no other metastore arguments are
      provided.
    </description>
  </property>

  <property>
    <name>sqoop.metastore.client.autoconnect.url</name>
    <value>jdbc:hsqldb:file:/tmp/sqoop-meta/meta.db;shutdown=true</value>
    <description>The connect string to use when connecting to a
      job-management metastore. If unspecified, uses ~/.sqoop/.
      You can specify a different path here.
    </description>
  </property>

  <property>
    <name>sqoop.metastore.client.autoconnect.username</name>
    <value>SA</value>
    <description>The username to bind to the metastore.
    </description>
  </property>

  <property>
    <name>sqoop.metastore.client.autoconnect.password</name>
    <value></value>
    <description>The password to bind to the metastore.
    </description>
  </property>

  <property>
    <name>sqoop.metastore.client.record.password</name>
    <value>true</value>
    <description>If true, allow saved passwords in the metastore.
    </description>
  </property>

  <property>
    <name>sqoop.metastore.server.location</name>
    <value>/tmp/sqoop-metastore/shared.db</value>
    <description>Path to the shared metastore database files.
    If this is not set, it will be placed in ~/.sqoop/.
    </description>
  </property>

  <property>
    <name>sqoop.metastore.server.port</name>
    <value>16000</value>
    <description>Port that this metastore should listen on.
    </description>
  </property>

</configuration>
  1. 修改configure-sqoop
vi bin/configure-sqoop

​ 将出现HCAT_HOME和ACCUMULO_HOME的判断逻辑注释掉.

[hadoop@hadoop100 sqoop-1.4.7]$ vi bin/configure-sqoop

 82 #if [ -z "${HCAT_HOME}" ]; then
 83 #  if [ -d "/usr/lib/hive-hcatalog" ]; then
 84 #    HCAT_HOME=/usr/lib/hive-hcatalog
 85 #  elif [ -d "/usr/lib/hcatalog" ]; then
 86 #    HCAT_HOME=/usr/lib/hcatalog
 87 #  else
 88 #    HCAT_HOME=${SQOOP_HOME}/../hive-hcatalog
 89 #    if [ ! -d ${HCAT_HOME} ]; then
 90 #       HCAT_HOME=${SQOOP_HOME}/../hcatalog
 91 #    fi
 92 #  fi
 93 #fi
 94 #if [ -z "${ACCUMULO_HOME}" ]; then
 95 #  if [ -d "/usr/lib/accumulo" ]; then
 96 #    ACCUMULO_HOME=/usr/lib/accumulo
 97 #  else
 98 #    ACCUMULO_HOME=${SQOOP_HOME}/../accumulo
 99 #  fi
100 #fi

134 ## Moved to be a runtime check in sqoop.
135 #if [ ! -d "${HCAT_HOME}" ]; then
136 #  echo "Warning: $HCAT_HOME does not exist! HCatalog jobs will fail."
137 #  echo 'Please set $HCAT_HOME to the root of your HCatalog installation.'
138 #fi
139 
140 #if [ ! -d "${ACCUMULO_HOME}" ]; then
141 #  echo "Warning: $ACCUMULO_HOME does not exist! Accumulo imports will fail."
142 #  echo 'Please set $ACCUMULO_HOME to the root of your Accumulo installation.'
143 #fi

这样做的目的是避免运行时出现下面的警告信息,其实也可以不用注释掉:

Warning: /opt/module/sqoop/bin/…/…/hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /opt/module/sqoop/bin/…/…/accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
  1. 添加所需的Jar包

​ 将MySQL的驱动(使用5.x版本,不要使用高版本的)上传到sqoop安装目录下的lib目录下

$ cp mysql-connector-java-5.1.44-bin.jar /opt/pkg/sqoop-1.4.7/lib/

​ 将$HIVE_HOME/lib/hive-common-3.1.2.jar拷贝或者软链接到$SQOOP_HOME/lib

$ ln -s /opt/pkg/hive/lib/hive-common-3.1.2.jar /opt/pkg/hbase/lib

​ 如果需要解析json,可下载java-json.har,放到sqoop目录下的lib里

$ cp java-json.jar /opt/pkg/sqoop-1.4.7/lib/

3、验证

sqoop version

[hadoop@hadoop100 zookeeper-3.4.11]$ sqoop-version 
21/01/11 00:48:43 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
Sqoop 1.4.7
git commit id 2328971411f57f0cb683dfb79d19d4d19d185dd8
Compiled by maugli on Thu Dec 21 15:59:58 STD 2017

4.异常处理

  1. sqoop导入hive表报错解决
Could not load org.apache.hadoop.hive.conf.HiveConf

​ 解决方法:

​ 只需把hive安装文件夹里的/lib/hive-common-x.x.x.jar复制出来放在sqoop安装文件夹的lib文件夹里就行的了

​ 注:上面的x.x.x意思不是这个jar包真的叫hive-common-x.x.x.jar,是表示版本号的,比如说我的hive版本是2.3.3的,所以我这个jar包名是hive-common-2.3.3.jar的。

  1. 查询导入报错
INFO mapreduce.ImportJobBase: Beginning query import.
Exception in thread "main" java.lang.NoClassDefFoundError: org/json/JSONObject

根据错误定位导缺少java-json jar包

然后去下载这个jar

http://www.java2s.com/Code/Jar/j/Downloadjavajsonjar.htm

把jar包拷贝到SQOOP_HOME/lib下面,然后重新运行SQOOP

Views: 46

08 – Hive 3.1.2 安装

官方手册

GettingStarted

Manual

1.安装及配置 Hive

(1)把 Hive 的安装包 apache-hive-3.1.2-bin.tar.gz 上传到 Linux 虚拟机的/opt/download目录下, 解压

$ wget https://mirrors.bfsu.edu.cn/apache/hive/hive-3.1.2/apache-hive-3.1.2-bin.tar.gz
$ tar -zxvf apache-hive-3.1.2-bin.tar.gz

(2)将解压的文件夹移动到/opt/pkg下,同时改名为hive

$ mv apache-hive-3.1.2 /opt/pkg/hive

(3)修改/etc/profile.d/hadoop.env.sh 文件,添加环境变量。

$ sudo vim /etc/profile.d/hadoop.env.sh 

添加以下内容。

# HIVE_HOME 3.1.2 
export HIVE_HOME=/opt/module/hive 
export PATH=PATH:HIVE_HOME/bin 

执行以下命令使环境变量生效。

$ source /etc/profile.d/hadoop.env.sh

(4)进到/opt/module/hive/lib 目录下执行以下命令,解决日志 jar 包冲突。

$ cd /opt/module/hive/lib
$ mv log4j-slf4j-impl-2.10.0.jar log4j-slf4j-impl-2.10.0.jar.bak

(5)解决guava版本低于hadoop版本的问题

$ cd /opt/module/hive/lib
$ mv guava-19.0.jar guava-19.0.jar.bak
$ ln -s /opt/pkg/hadoop/share/hadoop/common/lib/guava-27.0-jre.jar ./

删除或改名lib下较低版本的guava的jar包

从hadoop目录将guava的jar包拷贝或者软链接过来

如果不处理,初始化MySQL元数据的时候就会出错。

2.驱动复制

(1)将mysql-connector-java-5.1.27.tar 驱动包拷贝到Hive的lib目录。

$ cp mysql-connector-java-5.1.27.jar /opt/pkg/hive/lib/

3.配置

(1)在/opt/module/hive/conf 目录下创建一个 hive-site.xml 文件

$ touch hive-site.xml 
$ vim hive-site.xml 

(2)配置如下


[hadoop@hadoop100 ~]$ cat /opt/pkg/hive/conf/hive-site.xml
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>

  <!--配置 Hive 保存元数据信息所需的 MySQL URL 地址-->
  <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://hadoop100:3306/metastore?createDatabaseIfNotExist=true</value>
    <description>JDBC connect string for a JDBC metastore</description>
  </property>

  <!--配置 Hive 连接 MySQL 的驱动全类名-->
  <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.jdbc.Driver</value>
    <description>Driver class name for a JDBC metastore</description>
  </property>

  <!--配置 Hive 连接 MySQL 的用户名 -->
  <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>hive</value>
    <description>username to use against metastore database</description>
  </property>

  <!--配置 Hive 连接 MySQL 的密码 -->
  <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>hive1234</value>
    <description>password to use against metastore database</description>
  </property>

  <property>
    <name>hive.metastore.warehouse.dir</name>
    <value>/user/hive/warehouse</value>
  </property>

  <property>
    <name>hive.metastore.schema.verification</name>
    <value>false</value>
  </property>

   <!--Hive 元数据的 thrift 的 uri地址,可配多个 -->
  <property>
    <name>hive.metastore.uris</name>
    <value>thrift://hadoop100:9083</value>
  </property>

  <!--Hive server2 的 thrift 端口 -->
  <property>
    <name>hive.server2.thrift.port</name>
    <value>10000</value>
  </property>

  <property>
    <name>hive.server2.thrift.bind.host</name>
    <value>hadoop100</value>
  </property>

  <property>
    <name>hive.metastore.event.db.notification.api.auth</name>
    <value>false</value>
  </property>

  <property>
    <name>hive.cli.print.header</name>
    <value>true</value>
  </property>

  <property>
    <name>hive.cli.print.current.db</name>
    <value>true</value>
  </property>
</configuration>

配置详情见官方文档

4.初始化元数据库

(1)启动 MySQL。

[hadoop@hadoop100 ~]$mysql -hhadoop100 -P3306 -uhive -phive1234

(2)新建 Hive 元数据库(metastore)。

mysql> create database hive; 
mysql> quit; 

(3)初始化 Hive 元数据库。

$ schematool -initSchema -dbType mysql -verbose 

5.启动 Hive

(1)Hive 2.x 以上版本,要先启动 Metastore 和 Hiveserver2 服务,否则会报错。

FAILED: HiveException java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient 

(2)在/opt/pkg/hive/bin 目录下编写 Hive 服务启动脚本,在脚本中启动 Metastore 和 Hiveserver2 服务。

[hadoop@hadoop100 bin]$ vi /opt/pkg/hive/bin/hive-services.sh

#!/bin/bash
HIVE_LOG_DIR=$HIVE_HOME/logs

mkdir -p $HIVE_LOG_DIR

#检查进程是否运行正常,参数1为进程名,参数2为进程端口
function check_process()
{
    pid=$(ps -ef 2>/dev/null | grep -v grep | grep -i $1 | awk '{print $2}')
    ppid=$(netstat -nltp 2>/dev/null | grep $2 | awk '{print $7}' | cut -d '/' -f 1)
    echo $pid
    [[ "$pid" =~ "$ppid" ]] && [ "$ppid" ] && return 0 || return 1
}

function hive_start()
{
    metapid=$(check_process HiveMetastore 9083)
    cmd="nohup hive --service metastore >$HIVE_LOG_DIR/metastore.log 2>&1 &"
    cmd=$cmd" sleep 4; hdfs dfsadmin -safemode wait >/dev/null 2>&1"
    [ -z "$metapid" ] && eval $cmd || echo "Metastroe服务已启动"
    server2pid=$(check_process HiveServer2 10000)
    cmd="nohup hive --service hiveserver2 >$HIVE_LOG_DIR/hiveServer2.log 2>&1 &"
    [ -z "$server2pid" ] && eval $cmd || echo "HiveServer2服务已启动"
}

function hive_stop()
{
    metapid=$(check_process HiveMetastore 9083)
    [ "$metapid" ] && kill $metapid || echo "Metastore服务未启动"
    server2pid=$(check_process HiveServer2 10000)
    [ "$server2pid" ] && kill $server2pid || echo "HiveServer2服务未启动"
}

case $1 in
"start")
    hive_start
    ;;
"stop")
    hive_stop
    ;;
"restart")
    hive_stop
    sleep 2
    hive_start
    ;;
"status")
    check_process HiveMetastore 9083 >/dev/null && echo "Metastore服务运行正常" || echo "Metastore服务运行异常"
    check_process HiveServer2 10000 >/dev/null && echo "HiveServer2服务运行正常" || echo "HiveServer2服务运行异常"
    ;;
*)
    echo Invalid Args!
    echo 'Usage: '$(basename $0)' start|stop|restart|status'
    ;;
esac

(3)增加脚本执行权限。

$ sudo chmod +x hive-services.sh 

(4)启动 Hive 后台服务。

$ hive-services.sh start 

(5)查看 Hive 后台服务运行情况。

需要多试几次,因为服务从启动直到进程出现需要等待约一分钟左右时间。

$ hive-services.sh status 
Metastore 服务运行正常 
HiveServer2 服务运行正常 

(6)启动 Hive 客户端。

[hadoop@hadoop100 bin]$ hive
which: no hbase in (/opt/rh/llvm-toolset-7.0/root/usr/bin:/opt/pkg/hive/bin:/opt/pkg/flume/bin:/opt/pkg/kafka/bin:/opt/pkg/zookeeper/bin:/opt/pkg/hadoop/bin:/opt/pkg/hadoop/sbin:/opt/pkg/maven/bin:/sbin:/opt/pkg/java/bin:/opt/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/hadoop/.local/bin:/home/hadoop/bin)
Hive Session ID = 2b898c67-6677-4e98-8e23-0d70d358d206

Logging initialized using configuration in jar:file:/opt/pkg/hive/lib/hive-common-3.1.2.jar!/hive-log4j2.properties Async: true
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
Hive Session ID = 8aa334ad-253c-4974-bdb2-971b22bd3afb
hive (default)>

6. 日志

hive和beeline工具的日志默认在/tmp/<操作用户名>下生成,

如果希望执行命令时实时看到日志,也可以创建conf/hive-log4j2.properties以及conf/beeline-log4j2.properties(有模板)

  • 日志存放位置可以修改

    • property.hive.log.dir = /opt/pkg/hive/logs/
      property.hive.log.file = hive.log
  • 默认日志级别是info,会导致hive客户端中输出很多不必要的信息

    • 但是修改级别后并没有效果

hive服务器日志可以查看

$ tail -F /opt/pkg/hive/logs/hiveServer2.log

7. beeline 使用示例

Hive客户端工具后续将使用Beeline 替代HiveCLI ,并且后续版本也会废弃掉HiveCLI 客户端工具,Beeline是 Hive 0.11版本引入的新命令行客户端工具,它是基于SQLLine CLI的JDBC客户端。

beeline连接hiveserver2

beeline -u "jdbc:hive2://localhost:10000"  -nhadoop -p123123
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0
scan complete in 2ms
Connecting to jdbc:hive2://localhost:10000
Connected to: Apache Hive (version 2.3.4)
Driver: Hive JDBC (version 2.3.4)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 2.3.4 by Apache Hive
0: jdbc:hive2://localhost:10000>

类似hive-cli 的执行脚本功能

nohup beeline -u jdbc:hive2://127.0.0.1:10000  -n hadoop -p 123123  --color=true --silent=false  \
--hivevar p_date=${partitionDate} --hivevar f_date=${fileLocDate}  \
-f hdfs_add_partition_dmp_clearlog.hql  >> $logdir/load_${curDate}.log

beeline支持的操作

type command !help in beeline terminal

!help

output:

!addlocaldriverjar  Add driver jar file in the beeline client side.
!addlocaldrivername Add driver name that needs to be supported in the beeline
                    client side.
!all                Execute the specified SQL against all the current connections
!autocommit         Set autocommit mode on or off
!batch              Start or execute a batch of statements
!brief              Set verbose mode off
!call               Execute a callable statement
!close              Close the current connection to the database
!closeall           Close all current open connections
!columns            List all the columns for the specified table
!commit             Commit the current transaction (if autocommit is off)
!connect            Open a new connection to the database.
!dbinfo             Give metadata information about the database
!describe           Describe a table
!dropall            Drop all tables in the current database
!exportedkeys       List all the exported keys for the specified table
!go                 Select the current connection
!help               Print a summary of command usage
!history            Display the command history
!importedkeys       List all the imported keys for the specified table
!indexes            List all the indexes for the specified table
!isolation          Set the transaction isolation for this connection
!list               List the current connections
!manual             Display the BeeLine manual
!metadata           Obtain metadata information
!nativesql          Show the native SQL for the specified statement
!nullemptystring    Set to true to get historic behavior of printing null as
                    empty string. Default is false.
!outputformat       Set the output format for displaying results
                    (table,vertical,csv2,dsv,tsv2,xmlattrs,xmlelements, and
                    deprecated formats(csv, tsv))
!primarykeys        List all the primary keys for the specified table
!procedures         List all the procedures
!properties         Connect to the database specified in the properties file(s)
!quit               Exits the program
!reconnect          Reconnect to the database
!record             Record all output to the specified file
!rehash             Fetch table and column names for command completion
!rollback           Roll back the current transaction (if autocommit is off)
!run                Run a script from the specified file
!save               Save the current variabes and aliases
!scan               Scan for installed JDBC drivers
!script             Start saving a script to a file
!set                Set a beeline variable
!sh                 Execute a shell command
!sql                Execute a SQL command
!tables             List all the tables in the database
!typeinfo           Display the type map for the current connection
!verbose            Set verbose mode on

常用的几个command

  • !connect url –连接不同的Hive2服务器
  • !exit –退出shell
  • !help –显示全部命令列表
  • !verbose –显示查询追加的明细

5. 异常处理

一般hiverServer2的启动比较花时间,需等待5分钟左右才能启动,如果长时间还没有启动需要查看日志寻找失败原因

启动hiveserver2报错 1

Hive启动hiveserver2报错:Could not open client transport with JDBC Uri解决方案

报错信息:

Error: Could not open client transport with JDBC Uri: jdbc:hive2://node1:10000/hive_metadata;user=hadoop: java.net.ConnectException: 拒绝连接 (Connection refused) (state=08S01,code=0)
Beeline version 2.3.3 by Apache Hive

原因:hiveserver2增加了权限控制,需要在hadoop的配置文件中配置代理用户

解决方案

关闭dfs和yarn

$ stop-dfs.sh
$ stop-yarn.sh

在hadoop的core-site.xml中添加如下内容(其中hadoop为允许作为代理用户的用户及用户组)

<property>
   <name>hadoop.proxyuser.hadoop.hosts</name>
   <value>*</value>
</property>
<property>
   <name>hadoop.proxyuser.hadoop.groups</name>
   <value>*</value>
</property>

重启dfs

$ start-dfs.sh

重启yarn

$ start-yarn.sh

重新连接Hive

beeline -u jdbc:hive2://hadoop100:10000 -n hadoop -p

启动hiveserver2报错 2

若日志中发现如下异常信息:

Safe mode is ON. The reported blocks 3 needs additional 2 blocks to reach the threshold 0.9990 of total blocks 5. The number of live datanodes 2 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.

说明我们的损坏的文件比例超过了阈值, 这个阈值配置在hdfs中, 也就是说不允许任何一个块损坏掉. 如果我们配置成99%应该就不会触发safemode了.
由于系统断电,内存不足等原因导致dataNode丢失超过设置的丢失百分比,系统自动进入安全模式

解决办法

执行命令退出安全模式:

hadoop dfsadmin -safemode leave`

执行健康检查,删除损坏掉的block
hdfs fsck  /  -delete

运行后如果出现下面提示,表示修复完毕

The filesystem under path '/' is HEALTHY

如果没有修复则可以多执行一次hdfs fsck命令

beeline下无法导入hbase表

改为使用bin/hive就可以导入了, 可能是BUG。

Views: 63

07 Flume的安装与配置

安装

  1. 下载

    http://flume.apache.org/download.html

    http://archive.apache.org/dist/flume/stable/

    wget http://archive.apache.org/dist/flume/stable/apache-flume-1.9.0-bin.tar.gz

    这里使用最新的 apache-flume-1.9.0版本

  2. 解压安装

    tar zxvf apache-flume-1.9.0-bin.tar.gz -C /opt/pkg/
  3. 改目录名

    mv apache-flume-1.9.0-bin/ flume
  4. 配置环境变量,并让环境变量生效

    # FLUME 1.9.0
    export FLUME_HOME=/opt/pkg/flume
    export PATH=$FLUME_HOME/bin:$PATH
  5. 修改conf/flume-env.sh,配置JDK路径(该文件事先是不存在的,需要复制一份)
    复制:

    cp flume-env.template.sh flume-env.sh

    编辑文件,并设置如下内容:

    #设置JAVA_HOME:
    export JAVA_HOME = /opt/pkg/java         
    #修改默认的内存:  
    export JAVA_OPTS="-Xms1024m -Xmx1024m -Xss256k -Xmn2g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:-UseGCOverheadLimit"   
  6. 将hadoop-3.1.4安装路径下的依赖的jar软链接到flume-1.9.0/lib下:

    $ cd /opt/pkg/flume/lib
    $ ln -s /opt/pkg/hadoop/share/hadoop/common/hadoop-common-3.1.4.jar ./
    $ ln -s /opt/pkg/hadoop/share/hadoop/common/lib/commons-configuration2-2.1.1.jar ./
    $ ln -s /opt/pkg/hadoop/share/hadoop/common/lib/hadoop-auth-3.1.4.jar ./      
    $ ln -s /opt/pkg/hadoop/share/hadoop/common/lib/htrace-core4-4.1.0-incubating.jar ./
    $ ln -s /opt/pkg/hadoop/share/hadoop/common/lib/commons-io-2.5.jar ./         
    $ ln -s /opt/pkg/hadoop/share/hadoop/hdfs/hadoop-hdfs-3.1.4.jar ./
    

测试

  1. 验证

    bin/flume-ng version
    
    flume 1.9.0
    Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git
    Revision: d4fcab4f501d41597bc616921329a4339f73585e
    Compiled by fszabo on Mon Dec 17 20:45:25 CET 2018
    From source with checksum 35db629a3bda49d23e9b3690c80737f9
  2. 配置Flume HDFS Sink:
    在flume的conf目录新建一个log2hdfs.conf
    添加如下内容:

    [hadoop@hadoop100 conf]$ vi log2hdfs.conf
    
    # define the agent
    a1.sources=r1
    a1.channels=c1
    a1.sinks=k1
    
    # define the source
    #上传目录类型
    a1.sources.r1.type=spooldir
    a1.sources.r1.spoolDir=/tmp/flume-logs
    #定义自滚动日志完成后的后缀名
    a1.sources.r1.fileSuffix=.FINISHED
    #根据每行文本内容的大小自定义最大长度4096=4k
    a1.sources.r1.deserializer.maxLineLength=4096
    
    # define the sink
    a1.sinks.k1.type = hdfs
    #上传的文件保存在hdfs的/flume/logs目录下
    a1.sinks.k1.hdfs.path = hdfs://hadoop100:8020/flume/logs/%y-%m-%d/
    a1.sinks.k1.hdfs.filePrefix=access_log
    a1.sinks.k1.hdfs.fileSufix=.log
    a1.sinks.k1.hdfs.batchSize=1000
    a1.sinks.k1.hdfs.fileType = DataStream
    a1.sinks.k1.hdfs.writeFormat= Text
    # roll 滚动规则:按照数据块128M大小来控制文件的写入,与滚动相关其他的都设置成0
    #为了演示,这里设置成500k写入一次
    a1.sinks.k1.hdfs.rollSize= 512000
    a1.sinks.k1.hdfs.rollCount=0
    a1.sinks.k1.hdfs.rollInteval=0
    #控制生成目录的规则:一般是一天或者一周或者一个月一次,这里为了演示设置10秒
    a1.sinks.k1.hdfs.round=true
    a1.sinks.k1.hdfs.roundValue=10
    a1.sinks.k1.hdfs.roundUnit= second
    #是否使用本地时间
    a1.sinks.k1.hdfs.useLocalTimeStamp=true
    
    #define the channel
    a1.channels.c1.type = memory
    #自定义event的条数
    a1.channels.c1.capacity = 500000
    #flume事务控制所需要的缓存容量1000条event
    a1.channels.c1.transactionCapacity = 1000
    
    #source channel sink cooperation
    a1.sources.r1.channels = c1
    a1.sinks.k1.channel = c1
    

    注意

    • a1.sources.r1.spoolDir目录如果不存在需要先创建
    • a1.sinks.k1.hdfs.path目录会自动创建
      • 这里的路径是`hdfs://hadoop100:8020/flume/logs/yy-mm-dd/
      • 也就是每天的数据都会产生滚动日志。
      • 实际应该是按天或者按周、按月来生成滚动日志。
  3. 启动flume

  • 准备

    创建/tmp/flumn-logs, 并分配权限

     sudo mkdir -p  /tmp/flume-logs
  • 启动
    执行如下命令进行启动:

     flume-ng agent --conf ./conf/ -f ./conf/flume-hdfs.conf --name a1 -Dflume.root.logger=INFO,console
  • 测试

     # 每次执行追加一行文字
     echo xxxxxxxxxxxx >> /tmp/flume-logs/access_log123.log
  • 到Hadoop的控制台http://hadoop100:9870/查看 hdfs:hadoop100:9870/flume/logs/下有没有数据生成:

    image-20210110175013827

异常: 日志收集失败,报错:

2021-03-09 22:38:50,834 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:459)] process failed
java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V

原因: flume/lib/guava-xxx.jar 和 hadoop自带的jar包发生冲突

解决: 将flume/lib下的guava包删除或者改名, 只保留hadoop的版本即可

Views: 135

06 Centos7的MySQL安装

卸载mariaDb

mysql被oracle收购后为了防止mysql有可能变成闭源,因此mysql创始人maria就开源做了一个mariaDb, centos7是自带使用了这个数据库的.因此安装mysql之前,应当首先卸载mariadb数据库:

$ sudo yum list installed | grep mariadb    #检查mariadb是否已安装

$ sudo yum list installed | grep mariadb
mariadb-libs.x86_64                     1:5.5.56-2.el7                 @anaconda

$ sudo yum -y remove mariadb*    #全部卸载

同理如果是已经安装过其他版本的MySQL,安装新的版本之前也需要按照如上方法进行卸载。

安装MySQL

下载mysql的YUM源: https://dev.mysql.com/downloads/repo/yum/

YUM源的官方安装说明:https://dev.mysql.com/doc/mysql-yum-repo-quick-guide/en/

下载源mysql的YUM源,官网提供的是最新的MySQL8.0的源,内含其他较低版本的源

wget -P /home/hadoop/softwares https://dev.mysql.com/get/mysql80-community-release-el7-3.noarch.rpm  --no-check-certificate

由于我们是下载到/home/hadoop/softwares目录下,所以先切换到该目录下:

cd /home/hadoop/softwares

安装mysql的YUM源:

sudo rpm -ivh mysql80-community-release-el7-3.noarch.rpm

检查mysql的YUM源是否安装成功:

sudo yum repolist enabled | grep mysql

mysql-connectors-community/x86_64       MySQL Connectors Community           175
mysql-tools-community/x86_64            MySQL Tools Community                120
mysql80-community/x86_64                MySQL 8.0 Community Server           211

选择要启用的mysql版本

查看源里包含的mysql版本,执行:

sudo yum repolist all | grep mysql

mysql-cluster-7.5-community/x86_64 MySQL Cluster 7.5 Community      禁用
mysql-cluster-7.5-community-source MySQL Cluster 7.5 Community - So 禁用
mysql-cluster-7.6-community/x86_64 MySQL Cluster 7.6 Community      禁用
mysql-cluster-7.6-community-source MySQL Cluster 7.6 Community - So 禁用
mysql-cluster-8.0-community/x86_64 MySQL Cluster 8.0 Community      禁用
mysql-cluster-8.0-community-source MySQL Cluster 8.0 Community - So 禁用
mysql-connectors-community/x86_64  MySQL Connectors Community       启用:    175
mysql-connectors-community-source  MySQL Connectors Community - Sou 禁用
mysql-tools-community/x86_64       MySQL Tools Community            启用:    120
mysql-tools-community-source       MySQL Tools Community - Source   禁用
mysql-tools-preview/x86_64         MySQL Tools Preview              禁用
mysql-tools-preview-source         MySQL Tools Preview - Source     禁用
mysql55-community/x86_64           MySQL 5.5 Community Server       禁用
mysql55-community-source           MySQL 5.5 Community Server - Sou 禁用
mysql56-community/x86_64           MySQL 5.6 Community Server       禁用
mysql56-community-source           MySQL 5.6 Community Server - Sou 禁用
mysql57-community/x86_64           MySQL 5.7 Community Server       禁用
mysql57-community-source           MySQL 5.7 Community Server - Sou 禁用
mysql80-community/x86_64           MySQL 8.0 Community Server       启用:    211
mysql80-community-source           MySQL 8.0 Community Server - Sou 禁用

可以通过类似下面的语句来启动和禁用某些版本,比如这里是禁用默认启用的MySQL8.0的源,转而启用我们需要安装的MySQL5.7的源。

或者通过修改/etc/yum.repos.d/mysql-community.repo文件,改变默认安装的mysql版本。比如要安装5.7版本,将8.0源的enabled=1改成enabled=0,然后再将5.7源的enabled=0改成enabled=1即可。

sudo yum-config-manager --enable mysql57-community
sudo yum-config-manager --disable mysql80-community

要使用yum-config-manager命令, 需要先安装yum-utils: sudo yum install yum-utils

注意: 同一时间只允许enable一个MySQL版本。

如果只需要安装MySQL5.7的版本,也可以不看上面,直接本地安装指定的远程MySQL5.7源

yum localinstall https://dev.mysql.com/get/mysql57-community-release-el7-9.noarch.rpm

查看当前的启用的 MySQL 版本:

yum repolist enabled | grep mysql

mysql-connectors-community/x86_64       MySQL Connectors Community           175
mysql-tools-community/x86_64            MySQL Tools Community                120
mysql57-community/x86_64                MySQL 5.7 Community Server           464

安装MySQL

sudo yum install -y mysql-community-server

已加载插件:fastestmirror
mysql-connectors-community                                             | 2.6 kB  00:00:00     
mysql-tools-community                                                  | 2.6 kB  00:00:00     
mysql57-community                                                      | 2.6 kB  00:00:00     
mysql57-community/x86_64/prima FAILED                                          
http://repo.mysql.com/yum/mysql-5.7-community/el/7/x86_64/repodata/cae8c564a11e5fca11dbe958c273358f206f1bba-primary.sqlite.bz2: [Errno 14] curl#7 - "Failed connect to repo.mysql.com:80; Connection refused"
正在尝试其它镜像。
mysql57-community/x86_64/primary_db                                    | 247 kB  00:00:00     
Loading mirror speeds from cached hostfile
 * base: mirrors.aliyun.com
 * extras: mirrors.aliyun.com
 * updates: mirrors.aliyun.com
正在解决依赖关系
--> 正在检查事务
---> 软件包 mysql-community-server.x86_64.0.5.7.32-1.el7 将被 安装
--> 正在处理依赖关系 mysql-community-common(x86-64) = 5.7.32-1.el7,它被软件包 mysql-community-server-5.7.32-1.el7.x86_64 需要
--> 正在处理依赖关系 mysql-community-client(x86-64) >= 5.7.9,它被软件包 mysql-community-server-5.7.32-1.el7.x86_64 需要
--> 正在检查事务
---> 软件包 mysql-community-client.x86_64.0.5.7.32-1.el7 将被 安装
--> 正在处理依赖关系 mysql-community-libs(x86-64) >= 5.7.9,它被软件包 mysql-community-client-5.7.32-1.el7.x86_64 需要
---> 软件包 mysql-community-common.x86_64.0.5.7.32-1.el7 将被 安装
--> 正在检查事务
---> 软件包 mysql-community-libs.x86_64.0.5.7.32-1.el7 将被 安装
--> 解决依赖关系完成

依赖关系解决

==============================================================================================
 Package                      架构         版本                 源                       大小
==============================================================================================
正在安装:
 mysql-community-server       x86_64       5.7.32-1.el7         mysql57-community       173 M
为依赖而安装:
 mysql-community-client       x86_64       5.7.32-1.el7         mysql57-community        25 M
 mysql-community-common       x86_64       5.7.32-1.el7         mysql57-community       308 k
 mysql-community-libs         x86_64       5.7.32-1.el7         mysql57-community       2.3 M

事务概要
==============================================================================================
安装  1 软件包 (+3 依赖软件包)

总下载量:201 M
安装大小:875 M

....

已安装:
  mysql-community-server.x86_64 0:5.7.32-1.el7                                                

作为依赖被安装:
  mysql-community-client.x86_64 0:5.7.32-1.el7  
  mysql-community-common.x86_64 0:5.7.32-1.el7 
  mysql-community-libs.x86_64 0:5.7.32-1.el7 

启动MySQL服务

systemctl start mysqld.service # 或 service mysqld start

配置开机启动

systemctl enable mysqld.service

查看运行状态

systemctl status mysqld.service

mysql      2574      1  1 23:49 ?        00:00:00 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid

查看到进程信息

netstat -anpl  | grep mysql

查看端口,可以看出mysql server的进程mysqld所使用的默认端口即3306

$ sudo netstat -anpl | grep tcp
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1412/sshd         
tcp        0     52 192.168.186.103:22      192.168.186.1:54058     ESTABLISHED 2287/sshd: hadoop 
tcp6       0      0 :::3306                 :::*                    LISTEN      2574/mysqld       
tcp6       0      0 192.168.186.103:3888    :::*                    LISTEN      2060/java         
tcp6       0      0 :::22                   :::*                    LISTEN      1412/sshd         
tcp6       0      0 :::37791                :::*                    LISTEN      2060/java         
tcp6       0      0 :::2181                 :::*                    LISTEN      2060/java 

找到临时密码

但是要登陆mysql,我们需要root密码,这个密码是安装时随机生成在MySQL的服务器日志中的

grep "temporary password" /var/log/mysqld.log

2020-02-26T17:05:45.104999Z 1 [Note] A temporary password is generated for root@localhost: bl/!6qaU.wuX

登录MySQL

# 回车后输入在日志中找到的临时root登录密码登录
mysql -u roop -p 

问题: 日志里没有找到临时密码, 或者密码不能登录,可能的原因就是之前安装过MySQL

解决办法:

sudo rm -rf /var/lib/mysql

再重启服务后就可以在日志文件中找到重新生成的临时密码了

systemctl restart mysqld.service

$ cat /var/log/mysqld.log | grep "temporary password"
2020-02-26T17:05:45.104999Z 1 [Note] A temporary password is generated for root@localhost: bl/!6qaU.wuX
2020-08-01T16:07:26.941916Z 1 [Note] A temporary password is generated for root@localhost: x/ttqh=)_6Z/

也可以直接使用明文密码登陆,像这样

mysql -uroop -p"x/ttqh=)_6Z/"

初始密码我们可以这样操作,因为很快我们会将将其改为其他的密码。

但是正常使用时不建议这样 不安全

黑客可以使用history命令查看到你在命令行的输入获取到你的MySQL的root密码

另外MySQL报错在/var下面的log目录下可以查看


如果是虚拟机环境为了方便可以选择配置跳过密码验证

vim /etc/my.cnf

添加以下代码

default-authentication-plugin=mysql_native_password #配置文件本来就有去掉注释即可
symbolic-links=0
skip-grant-tables #跳过密码验证

第一次登陆后系统会很快提示你修改掉默认密码

mysql> show databases;
ERROR 1820 (HY000): You must reset your password using ALTER USER statement before executing this statement.

尝试修改密码(虚拟机可以设置简单密码,如果是购买的服务器则需设置复杂密码并修改默认端口号)

mysql> alter user 'root'@'localhost' identified by 'niit1234';
ERROR 1819 (HY000): Your password does not satisfy the current policy requirements

密码安全策略

但是基于密码策略,所设置的密码必须要包含大小写字母、数字和字符。

修改策略并设置简单密码

将策略要求置为0(LOW),长度要求置为1

mysql> set global validate_password_policy=0;
mysql> set global validate_password_length=1;

关于密码安全策略:

通过show命令查看当前的策略

mysql> show variables like 'validate_password%';

+--------------------------------------+--------+
| Variable_name                        | Value  |
+--------------------------------------+--------+
| validate_password_check_user_name    | OFF    |
| validate_password_dictionary_file    |        |
| validate_password_length             | 8      |
| validate_password_mixed_case_count   | 1      |
| validate_password_number_count       | 1      |
| validate_password_policy             | MEDIUM |
| validate_password_special_char_count | 1      |
+--------------------------------------+--------+
7 rows in set (0.00 sec)

1) validate_password_policy:密码安全策略,默认MEDIUM策略

策略 检查规则
0 or LOW Length
1 or MEDIUM Length; numeric, lowercase/uppercase, and special characters
2 or STRONG Length; numeric, lowercase/uppercase, and special characters; dictionary file

2)validate_password_dictionary_file:密码策略文件,策略为STRONG才需要

3)validate_password_length:密码最少长度

4)validate_password_mixed_case_count:大小写字符长度,至少1个

5)validate_password_number_count :数字至少1个

6)validate_password_special_char_count:特殊字符至少1个

这样以后就可以使用新密码登陆了

开启远程登陆

mysql命令的-h选项可以设置需要远程登陆mysql的服务器的ip地址

如果是本地的话可以使用127.0.0.1或者直接省略

但是远程访问因为安全原因默认是关闭的

mysql> select Host,User from user;
+-----------+---------------+
| Host      | User          |
+-----------+---------------+
| localhost | mysql.session |
| localhost | mysql.sys     |
| localhost | root          |
+-----------+---------------+
3 rows in set (0.00 sec)

这里可以看到host都是localhost,这就是因为这些账号只有本地访问的权限

我们需要将root的访问权限扩大都允许任意方式访问(包括远程访问)

GRANT ALL PRIVILEGES ON *.* TO 'root'@'%'IDENTIFIED BY 'niit1234' WITH GRANT OPTION;

也可以创建一个专门用来支持远程访问的单独用户

GRANT ALL PRIVILEGES ON *.* TO 'remote'@'%' IDENTIFIED BY 'niit1234' WITH GRANT OPTION; 

如果你想允许用户myuser从ip为192.168.1.6的主机连接到mysql服务器,并使用mypassword作为密码

若你想限制能连接到mysql服务器的客户端所在的IP地址为'192.168.1.6'

并使用mypassword作为密码,则命令如下:

GRANT ALL PRIVILEGES ON . TO 'mysqluser'@'192.168.1.6'IDENTIFIED BY 'mypassword' WITH GRANT OPTION;  

使修改生效

现在虽然修改了远程访问权限,但是还没有生效

我们需要刷新权限(或者重启服务),当然直接在mysql执行环境刷新权限是更好的方法

mysql>FLUSH PRIVILEGES;

然后我们可以试试使用navicat远程连接虚拟机中mysql试试

navicat虽然是收费的还是可以下载到破解的版本

最后一点

如果你的服务器开启了防火墙也可能导致远程无法连接

因为mysql默认使用3306端口,我们需要开放这个端口给mysql客户端连接

修改编码

默认安装的设置对中文的支持不好,一般需要修改成utf8,但这里我们修改成utf8mb4编码,因为它是uft8的超集,减少更多乱码的问题,也是推荐使用的编码,比如需要插入emoji字符在数据库中,utf8的显示就会出现问题,而utf8mb4则没有问题。

首先查看当前的编码:

mysql> show variables like '%char%';
+--------------------------------------+----------------------------+
| Variable_name                        | Value                      |
+--------------------------------------+----------------------------+
| character_set_client                 | utf8                       |
| character_set_connection             | utf8                       |
| character_set_database               | latin1                     |
| character_set_filesystem             | binary                     |
| character_set_results                | utf8                       |
| character_set_server                 | latin1                     |
| character_set_system                 | utf8                       |
| character_sets_dir                   | /usr/share/mysql/charsets/ |
| validate_password_special_char_count | 1                          |
+--------------------------------------+----------------------------+
9 rows in set (0.00 sec)

为了修改编码,我们需要修改mysql的核心配置文件如下:

[hadoop@hadoop000 download]$ sudo vim /etc/my.cnf

# For advice on how to change settings please see
# For advice on how to change settings please see
# http://dev.mysql.com/doc/refman/5.7/en/server-configuration-defaults.html

[mysqld]
#
# Remove leading # and set to the amount of RAM for the most important data
# cache in MySQL. Start at 70% of total RAM for dedicated server, else 10%.
# innodb_buffer_pool_size = 128M
#
# Remove leading # to turn on a very important data integrity option: logging
# changes to the binary log between backups.
# log_bin
#
# Remove leading # to set options mainly useful for reporting servers.
# The server defaults are faster for transactions and fast SELECTs.
# Adjust sizes as needed, experiment to find the optimal values.
# join_buffer_size = 128M
# sort_buffer_size = 2M
# read_rnd_buffer_size = 2M
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock

# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0

log-error=/var/log/mysqld.log

# 默认服务器内部操作字符集
character-set-server=utf8mb4
# 默认服务器内部操作字符集校对规则
collation-server=utf8mb4_general_ci
# 默认的存储引擎
default-storage-engine=InnoDB
# 初始化连接时设置以下字符集:
# character_set_client
# character_set_results
# character_set_connection
init_connect='set names utf8mb4'

[client]
default-character-set=utf8mb4

[mysql]
default-character-set=utf8mb4
  • 30行以下为手动添加的配置

验证

mysql> show variables like '%char%';
+--------------------------------------+----------------------------+
| Variable_name                        | Value                      |
+--------------------------------------+----------------------------+
| character_set_client                 | utf8mb4                    |
| character_set_connection             | utf8mb4                    |
| character_set_database               | utf8mb4                    |
| character_set_filesystem             | binary                     |
| character_set_results                | utf8mb4                    |
| character_set_server                 | utf8mb4                    |
| character_set_system                 | utf8                       |
| character_sets_dir                   | /usr/share/mysql/charsets/ |
| validate_password_special_char_count | 1                          |
+--------------------------------------+----------------------------+
9 rows in set (0.00 sec)
  • character_set_client 是指客户端发送过来的语句的编码
  • character_set_database 是指服务器内部使用的编码
  • character_set_connection 是指mysqld收到客户端的语句后,要转换到的编码
  • character_set_results 是指server执行语句后,返回给客户端的数据的编码
  • character_set_system是元数据编码,无需修改
  • character_set_filesystem是文件系统的编码,2进制存储最有效,不能修改

【END】

rpm-bundle的离线安装

下载rpm-bundle包


$ wget http://mirrors.163.com/mysql/Downloads/MySQL-5.7/mysql-5.7.33-1.el7.x86_64.rpm-bundle.tar
$  tar xvf mysql-5.7.33-1.el7.x86_64.rpm-bundle.tar
$  mkdir mysql-jars
$  mv mysql-comm*.rpm mysql-jars/
$  cd mysql-jars/
$ ls
mysql-community-client-5.7.33-1.el7.x86_64.rpm
mysql-community-common-5.7.33-1.el7.x86_64.rpm
mysql-community-libs-5.7.33-1.el7.x86_64.rpm
mysql-community-libs-compat-5.7.33-1.el7.x86_64.rpm
mysql-community-server-5.7.33-1.el7.x86_64.rpm

依次手动安装

sudo rpm -ivh mysql-community-common-5.7.33-1.el7.x86_64.rpm
sudo rpm -ivh mysql-community-libs-5.7.33-1.el7.x86_64.rpm
sudo rpm -ivh mysql-community-libs-compat-5.7.33-1.el7.x86_64.rpm
sudo rpm -ivh mysql-community-client-5.7.33-1.el7.x86_64.rpm
sudo rpm -ivh mysql-community-server-5.7.33-1.el7.x86_64.rpm

Views: 71

04 配置HBase伪分布式环境

如果使用Hadoop3.1.x,建议配合Hbase2使用,如HBase2.2.3。

Hadoop version support matrix

  • √ = Tested to be fully-functional
  • ! = Known to not be fully-functional, or there are CVEs so we drop the support in newer minor releases
  • x = Not tested, may/may-not function
HBase-1.4.x HBase-1.6.x HBase-1.7.x HBase-2.2.x HBase-2.3.x
Hadoop-2.7.0 x x x x x
Hadoop-2.7.1+ x x x x
Hadoop-2.8.[0-2] x x x x x
Hadoop-2.8.[3-4] ! x x x x
Hadoop-2.8.5+ ! x x
Hadoop-2.9.[0-1] x x x x x
Hadoop-2.9.2+ ! x x
Hadoop-2.10.x ! !
Hadoop-3.1.0 x x x x x
Hadoop-3.1.1+ x x x
Hadoop-3.2.x x x x

1、解压

[hadoop@hadoop100 hbase-2.2.3]$ tar -zxvf hbase-2.2.3-bin.tar.gz -C /opt/pkg/

2、环境变量

[hadoop@hadoop100 hbase-2.2.3]$ vi conf/hbase-env.sh
[hadoop@hadoop100 hbase-2.2.3]$ sudo vim /etc/profile.d/env.sh 

# JAVA_HOME
export JAVA_HOME=/opt/pkg/java
export PATH=$JAVA_HOME/bin:$PATH

# HADOOP_HOME
export HADOOP_HOME=/opt/pkg/hadoop-3.1.4
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

# HBASE_HOME
export HBASE_HOME=/opt/pkg/hbase-2.2.3
export PATH=$HBASE_HOME/bin:$PATH

最后使用source命令使配置生效

[hadoop@hadoop100 hbase-2.2.3]$ source /etc/profile.d/env.sh

3、配置

拷贝hadoop的hdfs-site.xml和core-site.xml软链接到hbase的conf目录下面


# core-site.xml
$ ln -s /opt/pkg/hadoop-2.7.3/etc/hadoop/hdfs-site.xml /opt/pkg/hbase-2.2.3/conf/hdfs-site.xml
# hdfs-site.xml
$ ln -s /opt/pkg/hadoop-2.7.3/etc/hadoop/core-site.xml /opt/pkg/hbase-2.2.3/conf/core-site.xml

修改hbase的conf/hbase-env.sh

# The java implementation to use.  Java 1.7+ required.
# export JAVA_HOME=/usr/java/jdk1.6.0/
export JAVA_HOME=/opt/pkg/java

# Configure PermSize. Only needed in JDK7. You can safely remove it for JDK8+
# export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS -XX:PermSize=128m -XX:MaxPermSize=128m"
# export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -XX:PermSize=128m -XX:MaxPermSize=128m"

# Tell HBase whether it should manage it's own instance of Zookeeper or not.
# export HBASE_MANAGES_ZK=true
  • 首先,需要修改JAVA_HOME为真实JDK路径

  • Configure PermSize.下面的2条export语句只是针对JDK7的优化,如果不用JDK7则可以删除或者注释掉

  • 默认情况,由HBase自行管理内置在其内的Zookeeper服务

    • 如果想要使用外部Zookeeper集群来进行管理,需要设置
    export HBASE_MANAGES_ZK=false

修改hbase的核心配置文件conf/hbase-site.xml

<configuration>
    <!-- Hbase在HDFS上的根目录 -->
    <property>
        <name>hbase.rootdir</name>
        <value>hdfs://hadoop100:8020/hbase</value>
    </property>
    <!-- HBase在ZooKeeper上的快照数据存储位置 -->
    <property>
        <name>hbase.zookeeper.property.dataDir</name>
        <value>/opt/pkg/hadoop/data/tmp/hbase-zkdata</value>
    </property>
    <!-- 是否以集群的方式运行 -->
    <property>
        <name>hbase.cluster.distributed</name>
        <value>true</value>
    </property>
    <!-- Zookeeper集群的所有主机,如有多个则逗号分隔 -->
    <property>
        <name>hbase.zookeeper.quorum</name>
        <value>hadoop100</value>
    </property>
</configuration>
  • hbase.rootdir是hbase存储在HDFS上的数据的根目录,里面存储着所有region的数据

  • 无论是伪分布式还是全分布式,hbase.cluster.distributed都需要设置true

    • 如果设成false,则hbase和zookeeper会在一个JVM进程里运行
  • hbase.zookeeper.quorum的主机名默认是localhost,应改为zookeeper集群中所有机器的主机名,多个主机名之间使用逗号分隔

  • zookeeper的端口默认都是2181,如果不是可以添加

问题:

java.lang.IllegalStateException: The procedure WAL relies on the ability to hsync for proper operation during component failures, but the underlying filesystem does not support doing so. Please check the config value of 'hbase.procedure.store.wal.use.hsync' to set the desired level of robustness and ensure the config value of 'hbase.wal.dir' points to a FileSystem mount that can provide it.

解决:

  • 在hbase-site.xml增加配置
<property>
    <name>hbase.unsafe.stream.capability.enforce</name>
    <value>false</value>
</property>

4、测试

首先启动hadoop的dfs相关进程

[hadoop@hadoop100 lib]$ start-dfs.sh 

等半分钟后启动Hbase

[hadoop@hadoop100 lib]$ start-hbase.sh 
hadoop100: starting zookeeper, logging to /opt/pkg/hbase-2.2.3/logs/hbase-hadoop-zookeeper-hadoop100.out
starting master, logging to /opt/pkg/hbase-2.2.3/logs/hbase-hadoop-master-hadoop100.out
starting regionserver, logging to /opt/pkg/hbase-2.2.3/logs/hbase-hadoop-1-regionserver-hadoop100.out

使用hbase shell测试


[hadoop@hadoop100 lib]$ hbase shell
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 2.2.3, r67592f3d062743907f8c5ae00dbbe1ae4f69e5af, Tue Oct 25 18:10:20 CDT 2020

hbase(main):001:0> status
1 active master, 0 backup masters, 1 servers, 0 dead, 2.0000 average load

hbase(main):002:0> list
TABLE                                                                                                   
0 row(s) in 0.0490 seconds

=> []

hbase(main):004:0> create 'wc','cf'
0 row(s) in 4.6740 seconds

=> Hbase::Table - wc
hbase(main):005:0> put 'wc','hello', 'cf:word', 'word'
0 row(s) in 0.2030 seconds

hbase(main):006:0> scan 'wc'
ROW                         COLUMN+CELL                                                                 
 hello                      column=cf:word, timestamp=1610209834376, value=word                         
1 row(s) in 0.0590 seconds

hbase(main):007:0> 

界面

hbase的Master管理界面地址: http://hadoop100:16010

image-20210110012902737

hbase的RegionServer管理界面地址: http://hadoop100:16301

image-20210110013140999

常见问题

注意:如果启动HBase Shell时遇到警告:

[hadoop@hadoop100 hbase-2.2.3]$ hbase shell
SLF4J: Class path contains multiple SLF4J bindings.

原因是hadoop和hbase的jar包冲突了,解决办法,将hbase/lib下面的相同slf4j-log4j12的jar包改名即可

[hadoop@hadoop100 lib]$ mv slf4j-log4j12-1.7.5.jar slf4j-log4j12-1.7.5.jar.backup

Views: 45

Index