Cloudera Manager 5.14.4离线安装
- 安装环境
本次安装一共使用6台服务器
硬件环境：/目录500G，数据目录500G
操作系统：CentOS7
相关应用：JDK、Mysql
操作用户：root
- 系统配置(每个节点)
hostnamectl set-hostname jkbigdata01
# cat /etc/hostname
jkbigdata01
修改所有服务器
重启服务器使host主机名生效
- 编辑/etc/hosts 文件，添加主机地址解析
# more /etc/hosts
192.168.10*.33 jkbigdata01
192.168.10*.34 jkbigdata02
192.168.10*.29 jkbigdata03
192.168.10*.35 jkbigdata04
192.168.10*.36 jkbigdata05
192.168.10*.32 jkbigdata06
- 关闭防火墙
# systemctl stop firewalld.service
# systemctl disable firewalld.service
# systemctl status firewalld.service
firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: man:firewalld(1)
- 编辑/etc/selinux/config文件，将SELINUX值改为disabled
# vim /etc/selinux/config
- 编辑/proc/sys/vm/swappiness文件
Cloudera 建议将 /proc/sys/vm/swappiness 设置为最大值 10，系统默认设置为 30。
# cat /proc/sys/vm/swappiness
临时修改
# sysctl vm.swappiness=10
永久修改
# echo 'vm.swappiness=10'>> /etc/sysctl.conf
- 编辑系统透明大页面压缩配置
# echo never > /sys/kernel/mm/transparent_hugepage/defrag
# echo never > /sys/kernel/mm/transparent_hugepage/enabled
然后将同一命令添加到 /etc/rc.local初始化脚本中。
- 设置ssh免秘钥登录(主节点)
实现name节点与各data节点无密码登录
# ssh-****** //一路回车到完成
# cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
# chmod 600 ~/.ssh/authorized_keys
# scp -r ~/.ssh/authorized_keys 主机名:~/.ssh/
- 安装JDK
# mkdir /usr/local/jdk/
# tar -zxvf /usr/local/jdk -zxvf /opt/jdk-8u172-linux-x64.tar.gz -C /usr/local/jdk
# echo "export JAVA_HOME=/usr/local/jdk" >> /etc/profile
# echo "export CLASS_PATH=$JAVA_HOME/lib:$JAVA_HOME/jre/lib" >> /etc/profile
# echo "export PATH=.:$JAVA_HOME/bin:$PATH" >> /etc/profile
# source /etc/profile
- 配置NTP服务
上传rpm依赖包，rpm安装
主节点配置
# vim /etc/ntp.conf
配置需要同步的NTP服务器，如果银行体制内机构，填写提供的ntp服务器即可。
server xxxxx prefer
从节点配置
# vim /etc/ntp.conf
# systemctl restart ntpd
# ntpdate 主节点IP
server jkbigdata01
fudge jkbigdata01 startum 10
服务器时间配置
# timedatectl set-timezone Asia/Shanghai
NTP服务启动
# systemctl start ntpd
# systemctl enable ntpd
NTP服务同步测试
#ntpstat
显示synchronised to Ntp Server(xxx.xx.x.x.) 字样为成功
异常处理
如遇到ntp配置正确，依旧无法显示正在同步字样，则进行如下处理
#ps –ef |grep ntp
#kill -9 pid
- 安装MySQL5.7 (主节点)
卸载系统自带的mariadb
#rpm -qa|grep mariadb
mariadb-libs-5.5.56-2.el7.x86_64
# rpm -e --nodeps mariadb-libs-5.5.56-2.el7.x86_64
安装Mysql
安装、解压mysql
#tar –xvf mysql-5.7.27-1.el7.x86_64.rpm-bundle.tar
# rpm –ivh mysql-community-common-5.7.27-1.el7.x86_64.rpm
# rpm –ivh mysql-community-libs-5.7. 27-1.el7.x86_64.rpm
# rpm –ivh mysql-community-clent-5.7. 27-1.el7.x86_64.rpm
# rpm –ivh mysql-community-server-5.7.27-1.el7.x86_64.rpm
# rpm –ivh mysql-community-libs-compat-5.7.27-1.el7.x86_64.rpm
# mysql_instatll_db --datadir=/var/lib/mysql
# groupadd mysql
# useradd mysql
# chown mysql:mysql /var/lib/mysql –R
# chmod +777 /var/lib/mysql –R
mysql服务启动
# srevice mysqld start
mysql服务测试
# service mysqld status
含有Active:actice(running)字样则为成功
mysql登录密码修改
#vi /etc/my.cnf
在[mysqld]下加入
skip-grant-tables
character_set_server=utf8
init_connect=’SET NAMES utf8’
重启mysql
#service mysqld restart
#mysql –uroot –p //一路回车
mysql> update mysql.user set authentication_string=password(‘123456’) where user=’root’;
mysql> quit
#vi /etc/my.cnf
注释skip-grant-tables
重启mysql服务
#mysql –uroot –p
创建CDH所用的数据库并创建用户
mysql>grant all privileges on *.* to [email protected]'%' identified by '123456' WITH GRANT OPTION;
mysql> grant all privileges on *.* to [email protected]'%' identified by '123456' WITH GRANT OPTION;
mysql> CREATE DATABASE cm CHARACTER SET 'utf8' COLLATE 'utf8_general_ci';
mysql> CREATE DATABASE cm_am CHARACTER SET 'utf8' COLLATE 'utf8_general_ci';
mysql> CREATE DATABASE cm_hive CHARACTER SET 'utf8' COLLATE 'utf8_general_ci';
mysql> CREATE DATABASE cm_hue CHARACTER SET 'utf8' COLLATE 'utf8_general_ci';
mysql> CREATE DATABASE cm_oozie CHARACTER SET 'utf8' COLLATE 'utf8_general_ci';
mysql> CREATE DATABASE cm_rm CHARACTER SET 'utf8' COLLATE 'utf8_general_ci';
mysql> CREATE DATABASE cm_sentry CHARACTER SET 'utf8' COLLATE 'utf8_general_ci';
mysql> flush privileges;
mysql> alter database cm_hue character set latin1;
mysql> show create database cm_hue;
设置mysql开机启动
#systemctl enable mysqld.service
- 安装Cloudera Manager
1、基础搭建
#cd /opt
# tar -xzvf cloudera-manager-centos7-cm5.14.1_x86_64.tar.gz
解压完毕后， opt 目录下会出现cm-5.14.1和cloudera两个目录。
分发到各个节点
#scp –r c* [email protected]:$PWD
2、创建cloudera-scm用户
# useradd --system --home=/opt/cm-5.14.1/run/cloudera-scm-server/ --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm
3、配置agent
# vi /opt/cm-5.14.1/etc/cloudera-scm-agent/config.ini
将 server_host 设置为主节点的主机名，将 server_port 设置为 7182：
server_host=jkbigdata01
4、添加mysql-connector驱动
# cp mysql-connector-java-5.1.34-bin.jar /opt/cm-5.14.1/share/cmf/lib/
5、创建Cloudera Manager数据库（主节点）
# /opt/cm-5.14.1/share/cmf/schema/scm_prepare_database.sh mysql –hbigdata01 –P 3306 –uroot –p123456 –scm-host jkbigdata01 scm scm scm_password
输出内容有显示successful 字样为成功
6、添加CDH包（主节点）
# mv CDH-5.14.0-1.cdh5.14.0.p0.3-el7.parcel /opt/cloudera/parcel-repo/
# mv CDH-5.14.0-1.cdh5.14.0.p0.3-el7.parcel.sha /opt/cloudera/parcel-repo/CDH-5.14.0-1.cdh5.14.0.p0.3-el7.parcel.sha
# mv manifest.json /opt/cloudera/parcel-repo/manifest.json
7、启动Server 和Agent
1）在所有节点执行以下命令
#/opt/cm-5.14.1/etc/init.d/cloudera-scm-agent start(启动Agent)
2）在bigdata01 节点（主节点）执行以下命令
#/opt/cm-5.14.1/etc/init.d/cloudera-scm-server start(启动Server)
8、登录Cloudera Manager
http://10.10.6*.22*:7180/cmf/login
默认账号密码为 admin/admin
9、安装Parcel
1、选择免费版本
2、添加集群
填写集群定义的服务器名称或者ip（包括安装CM的主机本身），点击搜索，即可加载出所有主机。全选所有主机，并继续。
3、选择CDH版本
4、安装CDH
在系统检测无误后，进入CDH安装
5、选择服务类型
选择自定义服务
6、安装完成后，进入系统
配置集群JAVA_HOME
配置SPARK2、KAFKA
# mv SPARK2_ON_YARN-2.2.0.cloudera2.jar /opt/cloudera/csd
#chown cloudera-scm:cloudera-scm /opt/cloudera/csd/SPARK2_ON_YARN-2.2.0.cloudera2.jar
# mv SPARK2-2.2.0.cloudera2-1.cdh5.12.0.p0.232957-el7.parcel /opt/cloudera/parcel-repo
# mv SPARK2-2.2.0.cloudera2-1.cdh5.12.0.p0.232957-el7.parcel.sha1 /opt/cloudera/parcel-repo/ SPARK2-2.2.0.cloudera2-1.cdh5.12.0.p0.232957-el7.parcel.sha
# mv manifest.json /opt/cloudera/parcel-repo
#mv KAFKA-3.1.0-1.3.1.0.p0.35-el7.parcel.sha1 /opt/cloudera/parcel-repo/KAFKA-3.1.0-1.3.1.0.p0.35-el7.parcel.sha
# mv manifest.json /opt/cloudera/parcel-repo
# /opt/cm-5.14.1/etc/init.d/cloudera-scm-server restart
# /opt/cm-5.14.1/etc/init.d/cloudera-scm-agent restart
1）下载、分配、**kafka
2）添加服务
3）安装Kafka
选择提供服务的主机
4）排除安装错误
切换到配置页面更改配置项
5）配置服务
6）保存配置重新启动完成
7）下载、分配、**spark2
8）添加服务
完成安装
9）配置优化设置
Spark2 :spark_deploy_mode 改为 cluster
Spark2: spark_kafka_version 改为 0.10
彻底完工。