CentOS7.3 - CDH5.13 集群搭建实战笔记

CDH5.13 集群搭建实战笔记 


一、准备工作,系统环境搭建
1、创建data目录,挂载磁盘
[[email protected] ~]# ls /data/ -al
total 4
drwxr-xr-x. 4 root root 34 Dec 20 10:07 .
dr-xr-xr-x. 18 root root 4096 Dec 20 10:14 ..
drwxr-xr-x 2 root root 6 Sep 21 11:15 disk01
drwxr-xr-x 2 root root 6 Sep 21 11:18 disk02
[[email protected] ~]# df -hl
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/cl-root 3.7T 4.7G 3.7T 1% /
devtmpfs 126G 0 126G 0% /dev
tmpfs 126G 84K 126G 1% /dev/shm
tmpfs 126G 9.6M 126G 1% /run
tmpfs 126G 0 126G 0% /sys/fs/cgroup
/dev/sdb1 3.7T 34M 3.7T 1% /data/disk01
/dev/sdc1 3.7T 34M 3.7T 1% /data/disk02
/dev/sda2 1016M 167M 850M 17% /boot
/dev/sda1 200M 9.5M 191M 5% /boot/efi
tmpfs 26G 12K 26G 1% /run/user/42
tmpfs 26G 0 26G 0% /run/user/0

2、修改hostname,并配置/etc/hosts 文件
使用nmtui命令修改hostname
[[email protected] ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

10.18.218.17 cdh5node1
10.18.218.5 cdh5node2
10.18.218.9 cdh5node3

3、配置主节点ssh免密码登录
[[email protected] ~]# ssh-****** -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
99:60:26:05:37:b7:f4:13:67:86:a2:8e:b5:71:e9:98 [email protected]
The key's randomart image is:
+--[ RSA 2048]----+
| ..+ o ..+ |
| o +.o.= |
| . +..oo |
| ++.oo . |
| + BS |
| . E . |
| |
| |
| |
+-----------------+
[[email protected] ~]# scp /root/.ssh/id_rsa.pub [email protected]:/root/.ssh/
The authenticity of host 'cdh5node2 (10.18.218.5)' can't be established.
ECDSA key fingerprint is 7d:9a:d2:1b:2e:39:3c:26:94:55:1b:15:7a:fa:63:0e.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'cdh5node2,10.18.218.5' (ECDSA) to the list of known hosts.
[email protected]'s password:
id_rsa.pub 100% 396 0.4KB/s 00:00
[[email protected] ~]# scp /root/.ssh/id_rsa.pub [email protected]:/root/.ssh/
The authenticity of host 'cdh5node3 (10.18.218.9)' can't be established.
ECDSA key fingerprint is 60:14:1f:42:fb:49:ba:1c:1b:a8:33:03:5c:45:81:51.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'cdh5node3,10.18.218.9' (ECDSA) to the list of known hosts.
[email protected]'s password:
id_rsa.pub 100% 396 0.4KB/s 00:00
Last login: Wed Dec 20 10:19:22 2017 from 10.18.218.87
logout
Connection to cdh5node2 closed.

在每台slave机器上执行
[[email protected] .ssh]# cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys
[[email protected] .ssh]# chmod 600 /root/.ssh/authorized_keys

4、安装jdk
首先删除CentOS自带的Open JDK
[[email protected] yum.repos.d]# yum remove java
java-1.7.0-openjdk-headless.x86_64 java-1.8.0-openjdk-headless.x86_64 javapackages-tools.noarch
java-1.7.0-openjdk.x86_64 java-1.8.0-openjdk.x86_64
[[email protected] yum.repos.d]# yum remove java-1.7.0-openjdk-headless.x86_64 java-1.8.0-openjdk-headless.x86_64 javapackages-tools.noarch java-1.7.0-openjdk.x86_64 java-1.8.0-openjdk.x86_64
Loaded plugins: fastestmirror, langpacks
Resolving Dependencies
--> Running transaction check
---> Package java-1.7.0-openjdk.x86_64 1:1.7.0.111-2.6.7.8.el7 will be erased
--> Processing Dependency: java >= 1.5 for package: jline-1.0-8.el7.noarch
---> Package java-1.7.0-openjdk-headless.x86_64 1:1.7.0.111-2.6.7.8.el7 will be erased
--> Processing Dependency: java-headless >= 1:1.6 for package: 1:libreoffice-core-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libjvm.so()(64bit) for package: 1:libreoffice-ure-5.0.6.2-3.el7.x86_64
---> Package java-1.8.0-openjdk.x86_64 1:1.8.0.102-4.b14.el7 will be erased
--> Processing Dependency: java-1.8.0-openjdk for package: icedtea-web-1.6.2-4.el7.x86_64
---> Package java-1.8.0-openjdk-headless.x86_64 1:1.8.0.102-4.b14.el7 will be erased
---> Package javapackages-tools.noarch 0:3.4.1-11.el7 will be erased
--> Processing Dependency: jpackage-utils for package: rhino-1.7R4-5.el7.noarch
--> Processing Dependency: jpackage-utils for package: rhino-1.7R4-5.el7.noarch
--> Processing Dependency: jpackage-utils for package: tagsoup-1.2.1-8.el7.noarch
--> Processing Dependency: jpackage-utils >= 1.6 for package: tagsoup-1.2.1-8.el7.noarch
--> Running transaction check
---> Package icedtea-web.x86_64 0:1.6.2-4.el7 will be erased
---> Package jline.noarch 0:1.0-8.el7 will be erased
---> Package libreoffice-core.x86_64 1:5.0.6.2-3.el7 will be erased
--> Processing Dependency: libavmedialo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libbasegfxlo.so()(64bit) for package: 1:libreoffice-pdfimport-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libbasegfxlo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libbasegfxlo.so()(64bit) for package: 1:libreoffice-graphicfilter-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libbasegfxlo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libclewlo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libcomphelper.so()(64bit) for package: 1:libreoffice-impress-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libcomphelper.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libcomphelper.so()(64bit) for package: 1:libreoffice-graphicfilter-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libcomphelper.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libdbtoolslo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libdbtoolslo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libdrawinglayerlo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libeditenglo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libeditenglo.so()(64bit) for package: 1:libreoffice-graphicfilter-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libeditenglo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libfilelo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libi18nlangtag.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libi18nlangtag.so()(64bit) for package: 1:libreoffice-graphicfilter-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libi18nlangtag.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libi18nutil.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libi18nutil.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libmsfilterlo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libmsfilterlo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libooxlo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libooxlo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libopencllo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libreoffice-core = 1:5.0.6.2-3.el7 for package: 1:libreoffice-draw-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libreoffice-core = 1:5.0.6.2-3.el7 for package: 1:libreoffice-pdfimport-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libreoffice-core = 1:5.0.6.2-3.el7 for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libreoffice-core = 1:5.0.6.2-3.el7 for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libreoffice-core = 1:5.0.6.2-3.el7 for package: 1:libreoffice-pyuno-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libreoffice-core = 1:5.0.6.2-3.el7 for package: 1:libreoffice-graphicfilter-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libreoffice-core = 1:5.0.6.2-3.el7 for package: 1:libreoffice-impress-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libreoffice-core for package: unoconv-0.6-7.el7.noarch
--> Processing Dependency: libreoffice-core = 1:5.0.6.2-3.el7 for package: 1:libreoffice-langpack-en-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsaxlo.so()(64bit) for package: 1:libreoffice-impress-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsaxlo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsaxlo.so()(64bit) for package: 1:libreoffice-graphicfilter-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsaxlo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsblo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsblo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsfxlo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsfxlo.so()(64bit) for package: 1:libreoffice-graphicfilter-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsfxlo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsotlo.so()(64bit) for package: 1:libreoffice-impress-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsotlo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsotlo.so()(64bit) for package: 1:libreoffice-graphicfilter-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsotlo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsvllo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsvllo.so()(64bit) for package: 1:libreoffice-graphicfilter-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsvllo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsvtlo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsvtlo.so()(64bit) for package: 1:libreoffice-graphicfilter-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsvtlo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsvxcorelo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsvxcorelo.so()(64bit) for package: 1:libreoffice-graphicfilter-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsvxcorelo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsvxlo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libsvxlo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libswlo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libtklo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libtklo.so()(64bit) for package: 1:libreoffice-graphicfilter-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libtklo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libtllo.so()(64bit) for package: 1:libreoffice-impress-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libtllo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libtllo.so()(64bit) for package: 1:libreoffice-graphicfilter-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libtllo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libucbhelper.so()(64bit) for package: 1:libreoffice-impress-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libucbhelper.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libucbhelper.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libutllo.so()(64bit) for package: 1:libreoffice-impress-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libutllo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libutllo.so()(64bit) for package: 1:libreoffice-graphicfilter-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libutllo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libvbahelperlo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libvbahelperlo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libvcllo.so()(64bit) for package: 1:libreoffice-impress-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libvcllo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libvcllo.so()(64bit) for package: 1:libreoffice-graphicfilter-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libvcllo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libvcllo.so()(64bit) for package: 1:libreoffice-pdfimport-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libwriterperfectlo.so()(64bit) for package: 1:libreoffice-impress-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libwriterperfectlo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libwriterperfectlo.so()(64bit) for package: 1:libreoffice-graphicfilter-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libwriterperfectlo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libxolo.so()(64bit) for package: 1:libreoffice-impress-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libxolo.so()(64bit) for package: 1:libreoffice-calc-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libxolo.so()(64bit) for package: 1:libreoffice-graphicfilter-5.0.6.2-3.el7.x86_64
--> Processing Dependency: libxolo.so()(64bit) for package: 1:libreoffice-writer-5.0.6.2-3.el7.x86_64
---> Package libreoffice-ure.x86_64 1:5.0.6.2-3.el7 will be erased
---> Package rhino.noarch 0:1.7R4-5.el7 will be erased
---> Package tagsoup.noarch 0:1.2.1-8.el7 will be erased
--> Running transaction check
---> Package libreoffice-calc.x86_64 1:5.0.6.2-3.el7 will be erased
---> Package libreoffice-draw.x86_64 1:5.0.6.2-3.el7 will be erased
---> Package libreoffice-graphicfilter.x86_64 1:5.0.6.2-3.el7 will be erased
---> Package libreoffice-impress.x86_64 1:5.0.6.2-3.el7 will be erased
---> Package libreoffice-langpack-en.x86_64 1:5.0.6.2-3.el7 will be erased
---> Package libreoffice-pdfimport.x86_64 1:5.0.6.2-3.el7 will be erased
---> Package libreoffice-pyuno.x86_64 1:5.0.6.2-3.el7 will be erased
---> Package libreoffice-writer.x86_64 1:5.0.6.2-3.el7 will be erased
---> Package unoconv.noarch 0:0.6-7.el7 will be erased
--> Finished Dependency Resolution
--> Running transaction check
---> Package libreoffice-langpack-en.x86_64 1:5.0.6.2-3.el7 will be erased
---> Package libreoffice-langpack-en.x86_64 1:5.0.6.2-3.el7 will be erased
--> Finished Dependency Resolution

Dependencies Resolved

======================================================================================================================================================
Package Arch Version Repository Size
======================================================================================================================================================
Removing:
java-1.7.0-openjdk x86_64 1:1.7.0.111-2.6.7.8.el7 @anaconda 549 k
java-1.7.0-openjdk-headless x86_64 1:1.7.0.111-2.6.7.8.el7 @anaconda 91 M
java-1.8.0-openjdk x86_64 1:1.8.0.102-4.b14.el7 @anaconda 501 k
java-1.8.0-openjdk-headless x86_64 1:1.8.0.102-4.b14.el7 @anaconda 102 M
javapackages-tools noarch 3.4.1-11.el7 @anaconda 156 k
libreoffice-langpack-en x86_64 1:5.0.6.2-3.el7 @anaconda 0.0
Removing for dependencies:
icedtea-web x86_64 1.6.2-4.el7 @anaconda 2.1 M
jline noarch 1.0-8.el7 @anaconda 72 k
libreoffice-calc x86_64 1:5.0.6.2-3.el7 @anaconda 29 M
libreoffice-core x86_64 1:5.0.6.2-3.el7 @anaconda 238 M
libreoffice-draw x86_64 1:5.0.6.2-3.el7 @anaconda 2.2 M
libreoffice-graphicfilter x86_64 1:5.0.6.2-3.el7 @anaconda 1.3 M
libreoffice-impress x86_64 1:5.0.6.2-3.el7 @anaconda 4.8 M
libreoffice-pdfimport x86_64 1:5.0.6.2-3.el7 @anaconda 545 k
libreoffice-pyuno x86_64 1:5.0.6.2-3.el7 @anaconda 1.9 M
libreoffice-ure x86_64 1:5.0.6.2-3.el7 @anaconda 6.8 M
libreoffice-writer x86_64 1:5.0.6.2-3.el7 @anaconda 15 M
rhino noarch 1.7R4-5.el7 @anaconda 1.0 M
tagsoup noarch 1.2.1-8.el7 @anaconda 140 k
unoconv noarch 0.6-7.el7 @anaconda 196 k

Transaction Summary
======================================================================================================================================================
Remove 6 Packages (+14 Dependent packages)

Installed size: 497 M
Is this ok [y/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Erasing : unoconv-0.6-7.el7.noarch 1/20
Erasing : 1:libreoffice-calc-5.0.6.2-3.el7.x86_64 2/20
Erasing : 1:libreoffice-impress-5.0.6.2-3.el7.x86_64 3/20
Erasing : 1:libreoffice-writer-5.0.6.2-3.el7.x86_64 4/20
Erasing : 1:libreoffice-draw-5.0.6.2-3.el7.x86_64 5/20
Erasing : 1:libreoffice-pdfimport-5.0.6.2-3.el7.x86_64 6/20
Erasing : 1:libreoffice-graphicfilter-5.0.6.2-3.el7.x86_64 7/20
Erasing : icedtea-web-1.6.2-4.el7.x86_64 8/20
Erasing : rhino-1.7R4-5.el7.noarch 9/20
Erasing : jline-1.0-8.el7.noarch 10/20
Erasing : tagsoup-1.2.1-8.el7.noarch 11/20
Erasing : 1:libreoffice-langpack-en-5.0.6.2-3.el7.x86_64 12/20
Erasing : 1:java-1.8.0-openjdk-1.8.0.102-4.b14.el7.x86_64 13/20
Erasing : 1:java-1.8.0-openjdk-headless-1.8.0.102-4.b14.el7.x86_64 14/20
Erasing : 1:libreoffice-pyuno-5.0.6.2-3.el7.x86_64 15/20
Erasing : 1:libreoffice-core-5.0.6.2-3.el7.x86_64 16/20
Erasing : 1:libreoffice-ure-5.0.6.2-3.el7.x86_64 17/20
Erasing : 1:java-1.7.0-openjdk-1.7.0.111-2.6.7.8.el7.x86_64 18/20
Erasing : 1:java-1.7.0-openjdk-headless-1.7.0.111-2.6.7.8.el7.x86_64 19/20
Erasing : javapackages-tools-3.4.1-11.el7.noarch 20/20
Verifying : 1:libreoffice-writer-5.0.6.2-3.el7.x86_64 1/20
Verifying : rhino-1.7R4-5.el7.noarch 2/20
Verifying : 1:java-1.8.0-openjdk-1.8.0.102-4.b14.el7.x86_64 3/20
Verifying : 1:java-1.7.0-openjdk-headless-1.7.0.111-2.6.7.8.el7.x86_64 4/20
Verifying : javapackages-tools-3.4.1-11.el7.noarch 5/20
Verifying : icedtea-web-1.6.2-4.el7.x86_64 6/20
Verifying : 1:libreoffice-pdfimport-5.0.6.2-3.el7.x86_64 7/20
Verifying : 1:libreoffice-core-5.0.6.2-3.el7.x86_64 8/20
Verifying : 1:libreoffice-calc-5.0.6.2-3.el7.x86_64 9/20
Verifying : tagsoup-1.2.1-8.el7.noarch 10/20
Verifying : 1:libreoffice-graphicfilter-5.0.6.2-3.el7.x86_64 11/20
Verifying : 1:libreoffice-pyuno-5.0.6.2-3.el7.x86_64 12/20
Verifying : unoconv-0.6-7.el7.noarch 13/20
Verifying : jline-1.0-8.el7.noarch 14/20
Verifying : 1:libreoffice-draw-5.0.6.2-3.el7.x86_64 15/20
Verifying : 1:libreoffice-ure-5.0.6.2-3.el7.x86_64 16/20
Verifying : 1:libreoffice-impress-5.0.6.2-3.el7.x86_64 17/20
Verifying : 1:java-1.8.0-openjdk-headless-1.8.0.102-4.b14.el7.x86_64 18/20
Verifying : 1:java-1.7.0-openjdk-1.7.0.111-2.6.7.8.el7.x86_64 19/20
Verifying : 1:libreoffice-langpack-en-5.0.6.2-3.el7.x86_64 20/20

Removed:
java-1.7.0-openjdk.x86_64 1:1.7.0.111-2.6.7.8.el7 java-1.7.0-openjdk-headless.x86_64 1:1.7.0.111-2.6.7.8.el7
java-1.8.0-openjdk.x86_64 1:1.8.0.102-4.b14.el7 java-1.8.0-openjdk-headless.x86_64 1:1.8.0.102-4.b14.el7
javapackages-tools.noarch 0:3.4.1-11.el7 libreoffice-langpack-en.x86_64 1:5.0.6.2-3.el7

Dependency Removed:
icedtea-web.x86_64 0:1.6.2-4.el7 jline.noarch 0:1.0-8.el7 libreoffice-calc.x86_64 1:5.0.6.2-3.el7
libreoffice-core.x86_64 1:5.0.6.2-3.el7 libreoffice-draw.x86_64 1:5.0.6.2-3.el7 libreoffice-graphicfilter.x86_64 1:5.0.6.2-3.el7
libreoffice-impress.x86_64 1:5.0.6.2-3.el7 libreoffice-pdfimport.x86_64 1:5.0.6.2-3.el7 libreoffice-pyuno.x86_64 1:5.0.6.2-3.el7
libreoffice-ure.x86_64 1:5.0.6.2-3.el7 libreoffice-writer.x86_64 1:5.0.6.2-3.el7 rhino.noarch 0:1.7R4-5.el7
tagsoup.noarch 0:1.2.1-8.el7 unoconv.noarch 0:0.6-7.el7

Complete!

安装jdk
[[email protected] Software]# yum localinstall jdk-8u131-linux-x64.rpm
Loaded plugins: fastestmirror, langpacks
Examining jdk-8u131-linux-x64.rpm: 2000:jdk1.8.0_131-1.8.0_131-fcs.x86_64
Marking jdk-8u131-linux-x64.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package jdk1.8.0_131.x86_64 2000:1.8.0_131-fcs will be installed
--> Finished Dependency Resolution

Dependencies Resolved

======================================================================================================================================================
Package Arch Version Repository Size
======================================================================================================================================================
Installing:
jdk1.8.0_131 x86_64 2000:1.8.0_131-fcs /jdk-8u131-linux-x64 269 M

Transaction Summary
======================================================================================================================================================
Install 1 Package

Total size: 269 M
Installed size: 269 M
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : 2000:jdk1.8.0_131-1.8.0_131-fcs.x86_64 1/1
Unpacking JAR files...
tools.jar...
plugin.jar...
javaws.jar...
deploy.jar...
rt.jar...
jsse.jar...
charsets.jar...
localedata.jar...
Verifying : 2000:jdk1.8.0_131-1.8.0_131-fcs.x86_64 1/1

Installed:
jdk1.8.0_131.x86_64 2000:1.8.0_131-fcs

Complete!
[[email protected] Software]#
[[email protected] Software]#

[[email protected] Software]# java -version
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

5、安装MySQL
在node1 和 node2 上安装 mysql
[[email protected] yum.repos.d]# yum install mysql-community-server.x86_64
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
Resolving Dependencies
--> Running transaction check
---> Package mysql-community-server.x86_64 0:5.6.37-2.el7 will be installed
--> Processing Dependency: mysql-community-common(x86-64) = 5.6.37-2.el7 for package: mysql-community-server-5.6.37-2.el7.x86_64
--> Processing Dependency: mysql-community-client(x86-64) >= 5.6.10 for package: mysql-community-server-5.6.37-2.el7.x86_64
--> Processing Dependency: perl(DBI) for package: mysql-community-server-5.6.37-2.el7.x86_64
--> Running transaction check
---> Package mysql-community-client.x86_64 0:5.6.37-2.el7 will be installed
--> Processing Dependency: mysql-community-libs(x86-64) >= 5.6.10 for package: mysql-community-client-5.6.37-2.el7.x86_64
---> Package mysql-community-common.x86_64 0:5.6.37-2.el7 will be installed
---> Package perl-DBI.x86_64 0:1.627-4.el7 will be installed
--> Processing Dependency: perl(RPC::PlServer) >= 0.2001 for package: perl-DBI-1.627-4.el7.x86_64
--> Processing Dependency: perl(RPC::PlClient) >= 0.2000 for package: perl-DBI-1.627-4.el7.x86_64
--> Running transaction check
---> Package mariadb-libs.x86_64 1:5.5.52-1.el7 will be obsoleted
---> Package mysql-community-libs.x86_64 0:5.6.37-2.el7 will be obsoleting
---> Package perl-PlRPC.noarch 0:0.2020-14.el7 will be installed
--> Processing Dependency: perl(Net::Daemon) >= 0.13 for package: perl-PlRPC-0.2020-14.el7.noarch
--> Processing Dependency: perl(Net::Daemon::Test) for package: perl-PlRPC-0.2020-14.el7.noarch
--> Processing Dependency: perl(Net::Daemon::Log) for package: perl-PlRPC-0.2020-14.el7.noarch
--> Processing Dependency: perl(Compress::Zlib) for package: perl-PlRPC-0.2020-14.el7.noarch
--> Running transaction check
---> Package perl-IO-Compress.noarch 0:2.061-2.el7 will be installed
--> Processing Dependency: perl(Compress::Raw::Zlib) >= 2.061 for package: perl-IO-Compress-2.061-2.el7.noarch
--> Processing Dependency: perl(Compress::Raw::Bzip2) >= 2.061 for package: perl-IO-Compress-2.061-2.el7.noarch
---> Package perl-Net-Daemon.noarch 0:0.48-5.el7 will be installed
--> Running transaction check
---> Package perl-Compress-Raw-Bzip2.x86_64 0:2.061-3.el7 will be installed
---> Package perl-Compress-Raw-Zlib.x86_64 1:2.061-4.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

======================================================================================================================================================
Package Arch Version Repository Size
======================================================================================================================================================
Installing:
mysql-community-libs x86_64 5.6.37-2.el7 5giot 2.0 M
replacing mariadb-libs.x86_64 1:5.5.52-1.el7
mysql-community-server x86_64 5.6.37-2.el7 5giot 59 M
Installing for dependencies:
mysql-community-client x86_64 5.6.37-2.el7 5giot 19 M
mysql-community-common x86_64 5.6.37-2.el7 5giot 257 k
perl-Compress-Raw-Bzip2 x86_64 2.061-3.el7 base 32 k
perl-Compress-Raw-Zlib x86_64 1:2.061-4.el7 base 57 k
perl-DBI x86_64 1.627-4.el7 base 802 k
perl-IO-Compress noarch 2.061-2.el7 base 260 k
perl-Net-Daemon noarch 0.48-5.el7 base 51 k
perl-PlRPC noarch 0.2020-14.el7 base 36 k

Transaction Summary
======================================================================================================================================================
Install 2 Packages (+8 Dependent packages)

Total download size: 82 M
Is this ok [y/d/N]: y
Downloading packages:
(1/10): mysql-community-common-5.6.37-2.el7.x86_64.rpm | 257 kB 00:00:00
(2/10): mysql-community-libs-5.6.37-2.el7.x86_64.rpm | 2.0 MB 00:00:00
(3/10): perl-Compress-Raw-Bzip2-2.061-3.el7.x86_64.rpm | 32 kB 00:00:00
(4/10): perl-Compress-Raw-Zlib-2.061-4.el7.x86_64.rpm | 57 kB 00:00:00
(5/10): perl-IO-Compress-2.061-2.el7.noarch.rpm | 260 kB 00:00:00
(6/10): perl-Net-Daemon-0.48-5.el7.noarch.rpm | 51 kB 00:00:00
(7/10): perl-PlRPC-0.2020-14.el7.noarch.rpm | 36 kB 00:00:00
(8/10): perl-DBI-1.627-4.el7.x86_64.rpm | 802 kB 00:00:00
(9/10): mysql-community-client-5.6.37-2.el7.x86_64.rpm | 19 MB 00:00:03
(10/10): mysql-community-server-5.6.37-2.el7.x86_64.rpm | 59 MB 00:00:07
------------------------------------------------------------------------------------------------------------------------------------------------------
Total 11 MB/s | 82 MB 00:00:07
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : mysql-community-common-5.6.37-2.el7.x86_64 1/11
Installing : mysql-community-libs-5.6.37-2.el7.x86_64 2/11
Installing : mysql-community-client-5.6.37-2.el7.x86_64 3/11
Installing : 1:perl-Compress-Raw-Zlib-2.061-4.el7.x86_64 4/11
Installing : perl-Net-Daemon-0.48-5.el7.noarch 5/11
Installing : perl-Compress-Raw-Bzip2-2.061-3.el7.x86_64 6/11
Installing : perl-IO-Compress-2.061-2.el7.noarch 7/11
Installing : perl-PlRPC-0.2020-14.el7.noarch 8/11
Installing : perl-DBI-1.627-4.el7.x86_64 9/11
Installing : mysql-community-server-5.6.37-2.el7.x86_64 10/11
Erasing : 1:mariadb-libs-5.5.52-1.el7.x86_64 11/11
Verifying : perl-Compress-Raw-Bzip2-2.061-3.el7.x86_64 1/11
Verifying : perl-Net-Daemon-0.48-5.el7.noarch 2/11
Verifying : perl-IO-Compress-2.061-2.el7.noarch 3/11
Verifying : mysql-community-libs-5.6.37-2.el7.x86_64 4/11
Verifying : mysql-community-common-5.6.37-2.el7.x86_64 5/11
Verifying : 1:perl-Compress-Raw-Zlib-2.061-4.el7.x86_64 6/11
Verifying : mysql-community-server-5.6.37-2.el7.x86_64 7/11
Verifying : perl-DBI-1.627-4.el7.x86_64 8/11
Verifying : mysql-community-client-5.6.37-2.el7.x86_64 9/11
Verifying : perl-PlRPC-0.2020-14.el7.noarch 10/11
Verifying : 1:mariadb-libs-5.5.52-1.el7.x86_64 11/11

Installed:
mysql-community-libs.x86_64 0:5.6.37-2.el7 mysql-community-server.x86_64 0:5.6.37-2.el7

Dependency Installed:
mysql-community-client.x86_64 0:5.6.37-2.el7 mysql-community-common.x86_64 0:5.6.37-2.el7 perl-Compress-Raw-Bzip2.x86_64 0:2.061-3.el7
perl-Compress-Raw-Zlib.x86_64 1:2.061-4.el7 perl-DBI.x86_64 0:1.627-4.el7 perl-IO-Compress.noarch 0:2.061-2.el7
perl-Net-Daemon.noarch 0:0.48-5.el7 perl-PlRPC.noarch 0:0.2020-14.el7

Replaced:
mariadb-libs.x86_64 1:5.5.52-1.el7

Complete!

配置MySQL 密码 :
[[email protected] yum.repos.d]# systemctl enable mysqld
[[email protected] yum.repos.d]# systemctl start mysqld
[[email protected] yum.repos.d]# systemctl status mysqld
● mysqld.service - MySQL Community Server
Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2017-12-20 10:49:16 CST; 1min 10s ago
Process: 13604 ExecStartPost=/usr/bin/mysql-systemd-start post (code=exited, status=0/SUCCESS)
Process: 13531 ExecStartPre=/usr/bin/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
Main PID: 13603 (mysqld_safe)
CGroup: /system.slice/mysqld.service
├─13603 /bin/sh /usr/bin/mysqld_safe --basedir=/usr
└─13770 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mysqld.log --...

Dec 20 10:49:15 cdh5node1 mysql-systemd-start[13531]: Alternatively you can run:
Dec 20 10:49:15 cdh5node1 mysql-systemd-start[13531]: /usr/bin/mysql_secure_installation
Dec 20 10:49:15 cdh5node1 mysql-systemd-start[13531]: which will also give you the option of removing the test
Dec 20 10:49:15 cdh5node1 mysql-systemd-start[13531]: databases and anonymous user created by default. This is
Dec 20 10:49:15 cdh5node1 mysql-systemd-start[13531]: strongly recommended for production servers.
Dec 20 10:49:15 cdh5node1 mysql-systemd-start[13531]: See the manual for more instructions.
Dec 20 10:49:15 cdh5node1 mysql-systemd-start[13531]: Please report any problems at http://bugs.mysql.com/
Dec 20 10:49:16 cdh5node1 mysqld_safe[13603]: 171220 10:49:16 mysqld_safe Logging to '/var/log/mysqld.log'.
Dec 20 10:49:16 cdh5node1 mysqld_safe[13603]: 171220 10:49:16 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
Dec 20 10:49:16 cdh5node1 systemd[1]: Started MySQL Community Server.
[[email protected] yum.repos.d]# mysql -uroot
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.6.37 MySQL Community Server (GPL)

Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> SET PASSWORD = PASSWORD('passw0rd');
Query OK, 0 rows affected (0.00 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

mysql> exit
Bye
[[email protected] yum.repos.d]# mysql -u root -ppassw0rd
Warning: Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 3


6、关闭防火墙和SELINUX
[[email protected] Software]# systemctl stop firewalld
[[email protected] Software]# systemctl disable firewalld
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
Removed symlink /etc/systemd/system/basic.target.wants/firewalld.service.
[[email protected] Software]# systemctl status firewalld.service
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: man:firewalld(1)

Dec 20 10:17:52 cdh5node2 systemd[1]: Starting firewalld - dynamic firewall daemon...
Dec 20 10:17:53 cdh5node2 systemd[1]: Started firewalld - dynamic firewall daemon.
Dec 20 10:53:12 cdh5node2 systemd[1]: Stopping firewalld - dynamic firewall daemon...
Dec 20 10:53:13 cdh5node2 systemd[1]: Stopped firewalld - dynamic firewall daemon.
[[email protected] Software]# getenforce
Enforcing
[[email protected] Software]# vim /etc/selinux/config
[[email protected] Software]# reboot

7、配置ntp服务
参考
yum install ntp
systemctl enable ntpd
systemctl start ntpd

主节点和外部同步,其余节点和主节点进行同步
主节点配置:
# Permit all access over the loopback interface. This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
restrict 127.0.0.1
restrict ::1

# Hosts on local network are less restricted.
#restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap

# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
# #加入允许内网IP访问,10.18.218.0这是网段 # Hosts on local network are less restricted
restrict 10.18.218.0 mask 255.255.255.0 nomodify notrap

server 101.201.72.121
server 202.108.6.95

restrict 101.201.72.121 nomodify notrap noquery
restrict 202.108.6.95 nomodify notrap noquery

server 127.127.1.0
fudge 127.127.1.0 stratum 10

#broadcast 192.168.1.255 autokey # broadcast server
#broadcastclient # broadcast client
#broadcast 224.0.1.1 autokey # multicast server
#multicastclient 224.0.1.1 # multicast client
#manycastserver 239.255.254.254 # manycast server
#manycastclient 239.255.254.254 autokey # manycast client

从节点配置:

# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.
restrict default nomodify notrap nopeer noquery

# Permit all access over the loopback interface. This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
restrict 127.0.0.1
restrict ::1

# Hosts on local network are less restricted.
#restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap

# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
server 10.18.218.17

restrict 10.18.218.17 nomodify notrap noquery ##一定要加,不然主无法修改从上面的时间

#broadcast 192.168.1.255 autokey # broadcast server
#broadcastclient # broadcast client
#broadcast 224.0.1.1 autokey # multicast server
#multicastclient 224.0.1.1 # multicast client
#manycastserver 239.255.254.254 # manycast server
#manycastclient 239.255.254.254 autokey # manycast client



8、搭建CDH Yum源
下载 cdh parcel 和 rpm 包
下载链接:
CDH - https://archive.cloudera.com/cdh5/parcels/

-rw-r--r--. 1 root root 1934231261 Dec 19 09:15 CDH-5.13.1-1.cdh5.13.1.p0.2-el7.parcel
-rw-r--r--. 1 root root 41 Dec 19 09:10 CDH-5.13.1-1.cdh5.13.1.p0.2-el7.parcel.sha1
drwxrwxr-x. 3 jenkins jenkins 121 Dec 7 06:32 cm
-rw-r--r--. 1 root root 951176044 Dec 19 09:13 cm5.13.1-centos7.tar.gz
-rw-r--r--. 1 root root 106 Dec 19 09:13 cm5.13.1-centos7.tar.gz.md5
-rw-r--r--. 1 root root 114 Dec 19 09:13 cm5.13.1-centos7.tar.gz.sha1
-rw-r--r--. 1 root root 73766 Dec 20 15:27 manifest.json

注意一定在下载parcel的时候,一定要下载目录下的manifest.json 文件 , 否则cm-manager认不到parcel !!!

[[email protected] yum.repos.d]# cat cloudera-cm.repo
[cloudera-cm]
# Packages for Cloudera's Distribution for cm, Version 5, on RedHat or CentOS 7 x86_64
name=Cloudera's Distribution for cm, Version 5
baseurl=http://10.18.218.14/bigdata/cm/5/
gpgkey = http://10.18.218.14/bigdata/cm/RPM-GPG-KEY-cloudera
gpgcheck = 1

[[email protected] yum.repos.d]# cat CentOS-ISO.repo
[base]
name=CentOS1611-ISO
baseurl=http://10.18.218.14/centos1611-iso/
enabled=1
gpgcheck=0
gpgkey=http://10.18.218.14/centos1611-iso/RPM-GPG-KEY-CentOS-7

二、开始部署CDH集群
1、安装cm-manager
Install the Cloudera Manager Server Packages

sudo yum install cloudera-manager-daemons cloudera-manager-server

安装MySQL JDBC Driver

配置cm 数据库
[[email protected] Software]# ls /usr/share/cmf/schema/scm_prepare_database.sh
/usr/share/cmf/schema/scm_prepare_database.sh
[[email protected] Software]# /usr/share/cmf/schema/scm_prepare_database.sh mysql -h cdh5node1 -uroot -ppassw0rd --scm-host cdh5node1 scm scm scm
JAVA_HOME=/usr/java/jdk1.8.0_131
Verifying that we can write to /etc/cloudera-scm-server
[ main] DbProvisioner ERROR Unable to find the MySQL JDBC driver. Please make sure that you have installed it as per instruction in the installation guide.
[ main] DbProvisioner ERROR Stack Trace:
java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)[:1.8.0_131]
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)[:1.8.0_131]
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)[:1.8.0_131]
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)[:1.8.0_131]
at java.lang.Class.forName0(Native Method)[:1.8.0_131]
at java.lang.Class.forName(Class.java:264)[:1.8.0_131]
at com.cloudera.enterprise.dbutil.DbProvisioner.executeSql(DbProvisioner.java:283)[db-common-5.13.1.jar:]
at com.cloudera.enterprise.dbutil.DbProvisioner.doMain(DbProvisioner.java:104)[db-common-5.13.1.jar:]
at com.cloudera.enterprise.dbutil.DbProvisioner.main(DbProvisioner.java:123)[db-common-5.13.1.jar:]
--> Error 1, giving up (use --force if you wish to ignore the error)
[[email protected] Software]# find / -name *mysql-connector-java*
find: ‘/run/user/42/gvfs’: Permission denied
/usr/share/cmf/cloudera-scm-telepub/jars/mysql-connector-java-5.1.15.jar
/usr/share/cmf/common_jars/mysql-connector-java-5.1.15.jar
[[email protected] Software]# sudo cp /usr/share/cmf/common_jars/mysql-connector-java-5.1.15.jar /usr/share/java/mysql-connector-java.jar
cp: cannot create regular file ‘/usr/share/java/mysql-connector-java.jar’: No such file or directory
[[email protected] Software]# mkdir /usr/share/java
[[email protected] Software]# mkdir /usr/share/java
mkdir: cannot create directory ‘/usr/share/java’: File exists
[[email protected] Software]# sudo cp /usr/share/cmf/common_jars/mysql-connector-java-5.1.15.jar /usr/share/java/mysql-connector-java.jar
[[email protected] Software]# /usr/share/cmf/schema/scm_prepare_database.sh mysql -h cdh5node1 -uroot -ppassw0rd --scm-host cdh5node1 scm scm scm
JAVA_HOME=/usr/java/jdk1.8.0_131
Verifying that we can write to /etc/cloudera-scm-server
[ main] DbProvisioner ERROR Exception when creating/dropping database with user 'root' and jdbc url 'jdbc:mysql://cdh5node1/?useUnicode=true&characterEncoding=UTF-8'
java.sql.SQLException: Access denied for user 'root'@'cdh5node1' (using password: YES)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1073)[mysql-connector-java.jar:]
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3603)[mysql-connector-java.jar:]
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3535)[mysql-connector-java.jar:]
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:934)[mysql-connector-java.jar:]
at com.mysql.jdbc.MysqlIO.secureAuth411(MysqlIO.java:4104)[mysql-connector-java.jar:]
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1299)[mysql-connector-java.jar:]
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2338)[mysql-connector-java.jar:]
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2371)[mysql-connector-java.jar:]
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2163)[mysql-connector-java.jar:]
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:794)[mysql-connector-java.jar:]
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)[mysql-connector-java.jar:]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)[:1.8.0_131]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)[:1.8.0_131]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)[:1.8.0_131]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)[:1.8.0_131]
at com.mysql.jdbc.Util.handleNewInstance(Util.java:407)[mysql-connector-java.jar:]
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:378)[mysql-connector-java.jar:]
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:305)[mysql-connector-java.jar:]
at java.sql.DriverManager.getConnection(DriverManager.java:664)[:1.8.0_131]
at java.sql.DriverManager.getConnection(DriverManager.java:247)[:1.8.0_131]
at com.cloudera.enterprise.dbutil.DbProvisioner.executeSql(DbProvisioner.java:296)[db-common-5.13.1.jar:]
at com.cloudera.enterprise.dbutil.DbProvisioner.doMain(DbProvisioner.java:104)[db-common-5.13.1.jar:]
at com.cloudera.enterprise.dbutil.DbProvisioner.main(DbProvisioner.java:123)[db-common-5.13.1.jar:]
[ main] DbProvisioner ERROR Stack Trace:
java.sql.SQLException: Access denied for user 'root'@'cdh5node1' (using password: YES)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1073)[mysql-connector-java.jar:]
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3603)[mysql-connector-java.jar:]
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3535)[mysql-connector-java.jar:]
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:934)[mysql-connector-java.jar:]
at com.mysql.jdbc.MysqlIO.secureAuth411(MysqlIO.java:4104)[mysql-connector-java.jar:]
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1299)[mysql-connector-java.jar:]
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2338)[mysql-connector-java.jar:]
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2371)[mysql-connector-java.jar:]
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2163)[mysql-connector-java.jar:]
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:794)[mysql-connector-java.jar:]
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)[mysql-connector-java.jar:]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)[:1.8.0_131]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)[:1.8.0_131]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)[:1.8.0_131]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)[:1.8.0_131]
at com.mysql.jdbc.Util.handleNewInstance(Util.java:407)[mysql-connector-java.jar:]
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:378)[mysql-connector-java.jar:]
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:305)[mysql-connector-java.jar:]
at java.sql.DriverManager.getConnection(DriverManager.java:664)[:1.8.0_131]
at java.sql.DriverManager.getConnection(DriverManager.java:247)[:1.8.0_131]
at com.cloudera.enterprise.dbutil.DbProvisioner.executeSql(DbProvisioner.java:296)[db-common-5.13.1.jar:]
at com.cloudera.enterprise.dbutil.DbProvisioner.doMain(DbProvisioner.java:104)[db-common-5.13.1.jar:]
at com.cloudera.enterprise.dbutil.DbProvisioner.main(DbProvisioner.java:123)[db-common-5.13.1.jar:]
--> Error 1, giving up (use --force if you wish to ignore the error)
[[email protected] Software]# mysql -uroot -ppassw0rd
Warning: Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 6
Server version: 5.6.37 MySQL Community Server (GPL)

Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> grant all on *.* to 'root'@'%' identified by 'passw0rd' with grant option;
Query OK, 0 rows affected (0.00 sec)

mysql> grant all on *.* to 'root'@'cdh5node1' identified by 'passw0rd' with grant option;
Query OK, 0 rows affected (0.00 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

mysql> exit
Bye
[[email protected] Software]# /usr/share/cmf/schema/scm_prepare_database.sh mysql -h cdh5node1 -uroot -ppassw0rd --scm-host cdh5node1 scm scm scm
JAVA_HOME=/usr/java/jdk1.8.0_131
Verifying that we can write to /etc/cloudera-scm-server
Creating SCM configuration file in /etc/cloudera-scm-server
Executing: /usr/java/jdk1.8.0_131/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
[ main] DbCommandExecutor INFO Successfully connected to database.
All done, your SCM database is configured correctly!

启动cm-server
Wait several minutes for the Cloudera Manager Server to start. To observe the startup process, runtail -f /var/log/cloudera-scm-server/cloudera-scm-server.log on the Cloudera Manager Server host. If the Cloudera Manager Server does not start, seeTroubleshooting Installation and Upgrade Problems.
In a web browser, enterhttp://Server host:7180, whereServer host is the FQDN or IP address of the
启动成功后,会看见以下日志输出:
2017-12-20 14:34:16,091 INFO WebServerImpl:org.mortbay.log: jetty-6.1.26.cloudera.4
2017-12-20 14:34:16,093 INFO WebServerImpl:org.mortbay.log: Started [email protected]:7180
2017-12-20 14:34:16,093 INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.
2017-12-20 14:34:16,184 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Finished constructing repo:2017-12-20T06:34:16.184Z
2017-12-20 14:34:19,980 INFO ScmActive-0:com.cloudera.server.cmf.components.ScmActive: ScmActive completed successfully.

2、开始安装服务
如果图片无法显示,参考有道云笔记
http://note.youdao.com/noteshare?id=fe62cc699a461d85518157382c1edb7b&sub=44F3AB43F15844DCAD1161468180A535

在浏览器中输入
10.18.218.17:7180
输入用户名admin,密码admin,添加主机后进入安装界面
CentOS7.3 - CDH5.13 集群搭建实战笔记

CentOS7.3 - CDH5.13 集群搭建实战笔记
CentOS7.3 - CDH5.13 集群搭建实战笔记
CentOS7.3 - CDH5.13 集群搭建实战笔记
CentOS7.3 - CDH5.13 集群搭建实战笔记

CentOS7.3 - CDH5.13 集群搭建实战笔记CentOS7.3 - CDH5.13 集群搭建实战笔记CentOS7.3 - CDH5.13 集群搭建实战笔记CentOS7.3 - CDH5.13 集群搭建实战笔记CentOS7.3 - CDH5.13 集群搭建实战笔记CentOS7.3 - CDH5.13 集群搭建实战笔记CentOS7.3 - CDH5.13 集群搭建实战笔记CentOS7.3 - CDH5.13 集群搭建实战笔记


CentOS7.3 - CDH5.13 集群搭建实战笔记

CentOS7.3 - CDH5.13 集群搭建实战笔记
[[email protected] parcel-repo]# echo never > /sys/kernel/mm/transparent_hugepage/defrag
[[email protected] parcel-repo]# echo never > /sys/kernel/mm/transparent_hugepage/enabled
[[email protected] parcel-repo]# echo 1> /proc/sys/vm/swappiness

[[email protected] ~]# cat /proc/sys/vm/swappiness
1

使用这两句话,可以解决cdh 检查主机报错问题


集群mysql配置
CentOS7.3 - CDH5.13 集群搭建实战笔记

CentOS7.3 - CDH5.13 集群搭建实战笔记
[[email protected] parcel-repo]# mysql -uroot -ppassw0rd
Warning: Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 366
Server version: 5.6.37 MySQL Community Server (GPL)

Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> create database oozie default character set utf8;
Query OK, 1 row affected (0.00 sec)

mysql> grant all privileges on oozie.* to 'oozie'@'localhost' identified by 'oozie';
Query OK, 0 rows affected (0.00 sec)

mysql> grant all privileges on oozie.* to 'oozie'@'%' identified by 'oozie';
Query OK, 0 rows affected (0.00 sec)

mysql>
mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

mysql> exit
Bye
[[email protected] parcel-repo]# mysql -uroot -ppassw0rd
Warning: Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 367
Server version: 5.6.37 MySQL Community Server (GPL)

Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> create database oozie default character set utf8;
ERROR 1007 (HY000): Can't create database 'oozie'; database exists
mysql> create database hive default character set utf8;
Query OK, 1 row affected (0.00 sec)

mysql> grant all privileges on hive.* to 'hive'@'localhost' identified by 'hive';
Query OK, 0 rows affected (0.00 sec)

mysql> grant all privileges on hive.* to 'hive'@'%' identified by 'hive';
Query OK, 0 rows affected (0.00 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

mysql> create database amon default character set utf8;
Query OK, 1 row affected (0.00 sec)

mysql> grant all privileges on amon.* to 'amon'@'localhost' identified by 'amon';
Query OK, 0 rows affected (0.00 sec)

mysql> grant all privileges on amon.* to 'amon'@'%' identified by 'amon';
Query OK, 0 rows affected (0.00 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.01 sec)

mysql> create database hue default character set utf8;
Query OK, 1 row affected (0.00 sec)

mysql> grant all privileges on hue.* to 'hue'@'localhost' identified by 'hue';
Query OK, 0 rows affected (0.00 sec)

mysql> grant all privileges on hue.* to 'hue'@'%' identified by 'hue';
Query OK, 0 rows affected (0.00 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

mysql> grant all privileges on hue.* to 'hue'@'cdh5node1' identified by 'hue';
Query OK, 0 rows affected (0.00 sec)

mysql> grant all privileges on amon.* to 'amon'@'cdh5node1' identified by 'amon';
Query OK, 0 rows affected (0.00 sec)

mysql> grant all privileges on oozie.* to 'oozie'@'cdh5node1' identified by 'oozie';
Query OK, 0 rows affected (0.00 sec)

mysql> grant all privileges on hive.* to 'hive'@'cdh5node1' identified by 'hive';
Query OK, 0 rows affected (0.00 sec)

mysql>

CentOS7.3 - CDH5.13 集群搭建实战笔记


CentOS7.3 - CDH5.13 集群搭建实战笔记

这个配置项没太弄明白,有一个DataNode Default Group 和 DataNode Group 1 ???
而且cdh可以自己识别到/data/disk01 和 /data/disk02 , 有点儿神奇
CentOS7.3 - CDH5.13 集群搭建实战笔记
CentOS7.3 - CDH5.13 集群搭建实战笔记

CentOS7.3 - CDH5.13 集群搭建实战笔记

CentOS7.3 - CDH5.13 集群搭建实战笔记
完成配置:
CentOS7.3 - CDH5.13 集群搭建实战笔记

CentOS7.3 - CDH5.13 集群搭建实战笔记
CentOS7.3 - CDH5.13 集群搭建实战笔记
登陆CDH
CentOS7.3 - CDH5.13 集群搭建实战笔记

CDH 主界面 :
CentOS7.3 - CDH5.13 集群搭建实战笔记
CentOS7.3 - CDH5.13 集群搭建实战笔记
CentOS7.3 - CDH5.13 集群搭建实战笔记


FAQ
1、Network Interface Speed Suppress...
The following network interface(s) appear to be operating at less than full speed: virbr0-nic.2 host network interface(s) appear to be operating at full speed.For 1 host network interface(s), the Cloudera Manager Agent could not determine the duplex mode or interface speed.

solved this problem by disabling virbr0 interface.

virsh net-destroy default
virsh net-undefine default

也可以使用
ethtool -s eno1 speed 1000 duplex full autoneg off
修改网卡的运行速度
把网卡的速度调整到1000MB/s 以后,又会报丢帧的错误
进到HP iLO里面看了一下,服务器上的网卡好像有三种
第一种没写速度
第二种谢了1G
第三种写了10G

目前机器之间对拷数据都只有十几兆的速度,明显没有达到千兆或者万兆网卡的速度,是不是真的是网卡没有选对 ? 需要进BIOS里面设置一下 ?

2、Hive Metastore 无法创建数据库问题
更新jdbc driver
MySQL version:
mysql Ver 14.14 Distrib 5.6.37, for Linux (x86_64) using EditLine wrapper
Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.

JDBC Driver:
mysql-connector-java-5.1.44-bin.jar

参考:

欢迎加微信(17864298197)讨论大数据相关的技术问题