基础环境配置

配置主机名和网络映射

1
2
vi /etc/hostname
vi /etc/hosts

确保集群内的节点之间网络通畅

防火墙与setlinux文件(每个节点都执行)

1
2
3
4
5
systemctl stop firewalld

systemctl disable firewalld.service

iptables -F

关闭selinux

1
vi /etc/selinux/config

将SELINUX=disabled

配置节点之间SSH免密登录

在主节点下执行：

1
2
3
cd ~/.ssh

ssh-keygen -t rsa

将公钥复制到各个节点的~/.ssh/authorized_keys中

禁用透明大页(每个节点都执行)

1
2
3
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never >> /etc/rc.local  	###为了防止重启机器后，上述配置不生效，将以上内容写入 /etc/rc.local

禁用交换空间(每个节点都执行)

1
2
echo "vm.swappiness=0"  >> /etc/sysctl.conf   (永久降低虚拟内存需求率)
sysctl -p                                     (使命令生效)

安装NTP服务

1
2
#下载(每个节点都安装)
yum install -y ntp

①配置主节点

1
vi /etc/ntp.conf

去掉注释，将地址改成网段地址

注释掉原有的server，添加以下内容

1
2
3
4
restrict 172.16.13.133 nomodify notrap nopeer noquery
server 127.127.1.0       ###在实际中可能会出现节点网络连接丢失的情况，这个时候我们可以以本地时间作为时间服务

fudge  127.127.1.0  stratum  10

②配置其他从节点

1
vi /etc/ntp.conf

需要配置主节点的地址

③启动ntp服务(所有节点)

1
2
systemctl start ntpd.service
systemctl enable ntpd.service

④验证ntp配置是否成功

1
ntpstat

如果出现以上信息说明配置成功

如果出现这种情况也是正常的，ntp服务器配置完毕后，需要等待5-10分钟才能与/etc/ntp.conf中配置的标准时间进行同步。

安装repo(主节点安装)

1
yum install -y createrepo

安装HTTP(主节点安装)

1
2
3
4
yum -y install httpd         ###就是用的刚配置的cenos7系统源
systemctl start httpd          (启动httpd服务)
systemctl status httpd         (查看httpd服务状态)
systemctl enable httpd         (将httpd服务加入到开启启动项中)

启动成功后可以浏览 http://ip/，如果可以看到以下界面说明是成功的。

创建yum源(主节点执行)

①创建目录

1
mkdir  /var/www/html/cm

②将rpm安装包放到cm目录下

③在/var/www/html/cm目录下执行命令

1
createrepo .  #创建本地yum仓库

④在/etc/yum.repos.d下面新建cm.repo文件

1
vi cm.repo
1
2
3
4
5
[Cloudera_Manager]
name=Cloudera Manager
baseurl=http://ip/cm
enabled=1
gpgcheck=0

⑤将.repo文件发送到其他节点机器下

1
2
scp /etc/yum.repos.d/cm.repo node2:/etc/yum.repos.d/
scp /etc/yum.repos.d/cm.repo node3:/etc/yum.repos.d/

⑥清空yum缓存(所有节点都执行)

1
2
3
yum clean all   	清空缓存信息
yum repolist 	查看yum仓库列表
yum makecache 	构建yum缓存

jdk安装及配置(所有节点)

安装jdk

1
yum -y install jdk    ###因为var/www/html/cm目录下已经有jdk的rpm包作为yum源，多台服务器可以同时安装

默认安装路径为/usr/java

配置jdk

1
vi /etc/profile

添加如下内容：

1
2
export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
export PATH=$PATH:$JAVA_HOME/bin

配置下发到其他节点

1
2
scp /etc/profile node2:/etc/
scp /etc/profile node3:/etc/

配置生效

1
source /etc/profile

安装及配置mysql

1
2
#mysql需要安装以下依赖
yum -y install libaio perl net-tools

①将mysql的rpm包上传到主节点上

②卸载系统自带的mariadb数据库

1
rpm -qa|grep mariadb | xargs -I {} rpm -e --nodeps {}

③安装mysql数据库

1
2
3
4
5
6
7
8
9
10
11
rpm -ivh mysql-community-common-5.7.27-1.el7.x86_64.rpm

rpm -ivh mysql-community-libs-5.7.27-1.el7.x86_64.rpm

rpm -ivh mysql-community-devel-5.7.27-1.el7.x86_64.rpm

rpm -ivh mysql-community-client-5.7.27-1.el7.x86_64.rpm

rpm -ivh mysql-community-server-5.7.27-1.el7.x86_64.rpm

rpm -ivh mysql-community-libs-compat-5.7.26-1.el7.x86_64.rpm

④开启数据库服务

1
2
3
4
5
6
#开启数据库服务
systemctl start mysqld
#添加到自启动服务
systemctl enable mysqld
#查看状态
systemctl status mysqld

⑤获取mysql密码

1
grep 'temporary password' /var/log/mysqld.log

⑥修改密码

1
2
3
4
5
6
7
mysql -uroot -p 获取到的密码

alter user root@localhost identified by '2wsx#EDC'; #密码修改为2wsx#EDC

update user set host = '%' where user ='root'; #允许远程连接

flush privileges;	#更新

⑦创建数据库并授权

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
mysql -uroot -p2wsx#EDC -e ‘create database metastore default character set utf8;’

mysql -u root --password=‘2wsx#EDC’ -e “create user ‘hive’@‘%’ identified by ‘2wsx#EDC’”;

mysql -u root --password=‘2wsx#EDC’ -e ‘create database hive default character set utf8’;

mysql -u root --password=‘2wsx#EDC’ -e “grant all privileges on hive.* to ‘hive’@‘%’”;

mysql -u root --password=‘2wsx#EDC’ -e “grant all privileges on metastore.* to ‘hive’@‘%’”;

mysql -u root --password=‘2wsx#EDC’ -e “create user ‘amon’@‘%’ identified by ‘2wsx#EDC’”;

mysql -u root --password=‘2wsx#EDC’ -e ‘create database amon default character set utf8’;

mysql -u root --password=‘2wsx#EDC’ -e “grant all privileges on amon.* to ‘amon’@‘%’”;

mysql -u root --password=‘2wsx#EDC’ -e “create user ‘rman’@‘%’ identified by ‘2wsx#EDC’”;

mysql -u root --password=‘2wsx#EDC’ -e ‘create database rman default character set utf8’;

mysql -u root --password=‘2wsx#EDC’ -e “grant all privileges on rman.* to ‘rman’@‘%’”;

mysql -u root --password=‘2wsx#EDC’ -e “create user ‘sentry’@‘%’ identified by ‘2wsx#EDC’”;

mysql -u root --password=‘2wsx#EDC’ -e ‘create database sentry default character set utf8’;

mysql -u root --password=‘2wsx#EDC’ -e “grant all privileges on sentry.* to ‘sentry’@‘%’”;

mysql -u root --password=‘2wsx#EDC’ -e “create user ‘nav’@‘%’ identified by ‘2wsx#EDC’”;

mysql -u root --password=‘2wsx#EDC’ -e ‘create database nav default character set utf8’;

mysql -u root --password=‘2wsx#EDC’ -e “grant all privileges on nav.* to ‘nav’@‘%’”;

mysql -u root --password=‘2wsx#EDC’ -e “create user ‘navms’@‘%’ identified by ‘2wsx#EDC’”;

mysql -u root --password=‘2wsx#EDC’ -e ‘create database navms default character set utf8’;

mysql -u root --password=‘2wsx#EDC’ -e “grant all privileges on navms.* to ‘navms’@‘%’”;

mysql -u root --password=‘2wsx#EDC’ -e “create user ‘cm’@‘%’ identified by ‘2wsx#EDC’”;

mysql -u root --password=‘2wsx#EDC’ -e ‘create database cm default character set utf8’;

mysql -u root --password=‘2wsx#EDC’ -e “grant all privileges on cm.* to ‘cm’@‘%’”;

mysql -u root --password=‘2wsx#EDC’ -e “create user ‘oos’@‘%’ identified by ‘2wsx#EDC’”;

mysql -u root --password=‘2wsx#EDC’ -e ‘create database oos default character set utf8’;

mysql -u root --password=‘2wsx#EDC’ -e “grant all privileges on oos.* to ‘oos’@‘%’”;

mysql -u root --password=‘2wsx#EDC’ -e “create user ‘hue’@‘%’ identified by ‘2wsx#EDC’”;

mysql -u root --password=‘2wsx#EDC’ -e ‘create database hue default character set utf8’;

mysql -u root --password=‘2wsx#EDC’ -e “grant all privileges on hue.* to ‘hue’@‘%’”;

⑧复制mysql-contannor.jar

复制mysql-connect到/usr/share/java/,如果没有该文件夹可以自行创建

1
cp mysql-connector-java-5.1.47.jar /usr/share/java/

安装CM server和Agent

1
2
3
4
5
#主节点安装
yum install -y cloudera-manager-daemons cloudera-manager-server   cloudera-manager-agent 

#从节点安装
yum install -y cloudera-manager-daemons cloudera-manager-agent

修改agent配置文件

分别修改mini1,mini2,mini3agent所在的节点的配置文件config.ini内容

1
2
vi /etc/cloudera-scm-agent/config.ini
#server_host=node1 （这里是以node1作为server节点，让agent感知server所在节点）

初始化数据库

1
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql -h localhost -uroot -proot --scm-host localhost scm root root

上传CDH服务包

上传以下三个CHD6大数据服务组件资源包 到【node1】的 /opt/cloudera/parcel-repo 目录下
CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel
CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.sha
manifest.json

启动Cloudera Manager

1
2
3
4
5
systemctl start cloudera-scm-server      (启动server)
systemctl status cloudera-scm-server     (查看server状态)
systemctl stop cloudera-scm-server         (停止server)
systemctl enable cloudera-scm-server     (开机启动server)
tail -500f /var/log/cloudera-scm-server/cloudera-scm-server.log (查看server启动日志)

使用netstat -nltp命令查看【node1】端口7180 是否启动成功，如果出现7180端口，说明启动成功

访问ip+7180端口即可查看页面

启动Cloudera Manager agent(所有节点)

1
2
3
4
systemctl start cloudera-scm-agent    (启动agent)
systemctl stop cloudera-scm-agent
systemctl enable cloudera-scm-agent   (开机启动agent) 
tail -500f /var/log/cloudera-scm-agent/cloudera-scm-agent.log  (查看agent启动日志)

如果看到9000和19001端口说明启动成功

软件安装

①选择第二个服务组合

②角色分配

③数据库设置

要先通过测试连接才可以继续

④审核更改

⑤安装

组件版本

序号 组件 版本
1 CDH 6.2.0
2 Hadoop 3.0.0
3 Hive 2.1.0
4 Hue 4.2.0
5 Kafka 2.1.0
6 Oozie 5.1.0
7 spark 2.4.0
8 Sqoop 1.4.7
9 ZooKeeper 3.4.5
10 Hbase 2.1.0

主机运行状态不良 / Cluster not found

退回上一步，然后执行命令

1
2
rm -f /var/lib/cloudera-scm-agent/cm_guid
systemctl restart cloudera-scm-agent

fail to format namenode

清空残留数据，重新初始化

1
2
3
4
5
6
7
8
#手动格式化namenode
hdfs namenode -format

#namenode节点
rm -rf /dfs/nn

#datanode节点
rm -rf /dfs/dn

hadoop报错：Permission denied: user=root

1、找到hdfs-site.xml 的 HDFS 服务高级配置代码段（安全阀）
2、添加设置如下，保存更改，重启hdfs。

dfs.permissions.enabled 的值设置为false