项目：Lvs+Keepalived+Nginx+Tomcat高可用集群
项目拓扑：
（注：当keepalived和lvs结合，只需在配置keepalived，指定VIP地址和节点，不需再写lvs的规则。lvs不支持请求的区分，但Nginx可以，针对HTML和jsp的访问，将请求分离，故在Nginx节点处分为两份）
项目环境：
|
系统类型 |
IP地址 |
主机名 |
所需软件 |
|
Centos 7.4 1708 64bit |
192.168.100.101 |
ld1.linuxfan.cn |
keepalived-1.2.13.tar.gz sendEmail-v1.56.tar.gz |
|
Centos 7.4 1708 64bit |
192.168.100.102 |
ld2.linuxfan.cn |
keepalived-1.2.13.tar.gz sendEmail-v1.56.tar.gz |
|
Centos 7.4 1708 64bit |
192.168.100.103 |
ng1.linuxfan.cn |
nginx-1.12.2.tar.gz rpcbind nfs |
|
Centos 7.4 1708 64bit |
192.168.100.104 |
ng2.linuxfan.cn |
nginx-1.12.2.tar.gz rpcbind nfs |
|
Centos 7.4 1708 64bit |
192.168.100.105 |
tm1.linuxfan.cn |
apache-tomcat-9.0.10.tar.gz jdk-8u171-linux-x64.tar.gz rpcbind nfs |
|
Centos 7.4 1708 64bit |
192.168.100.106 |
tm2.linuxfan.cn |
apache-tomcat-9.0.10.tar.gz jdk-8u171-linux-x64.tar.gz rpcbind nfs |
|
Centos 7.4 1708 64bit |
192.168.100.107 |
st.linuxfan.cn |
rpcbind nfs mariadb-server mysql |
实验重点：
1.概述：此架构中keepalived所起到的作用就是对lvs架构中的调度器进行热备份。至少包含两台热备的负载调度器，两台台web的节点服务器；
2.重点：LVS架构中需要通过ipvsadm工具来对ip_vs这个模块进行编写规则，使用keepalived+lvs时，不需要用到ipvsadm管理工具，不需要ipvsadm手动编写规则，用在keepalived的配置文件中指定配置项来将其取代
3.keepalived的节点健康检查：keepalived可以通过对real server的某个端口进行节点健康检查，来执行相应的操作，由notify_down配置项来完成
安装并配置后端两台tomcat（两台tomcat服务器配置相同，在此只列出其中一台配置）
[[email protected] ~]# ls
apache-tomcat-9.0.10.tar.gz jdk-8u171-linux-x64.tar.gz
[[email protected]~]# rpm -qa |grep java
[[email protected] ~]# tar zxvf jdk-8u171-linux-x64.tar.gz
[[email protected] ~]# mv jdk1.8.0_171/ /usr/local/java
[[email protected] ~]# ls /usr/local/java
bin db javafx-src.zip lib man release THIRDPARTYLICENSEREADME-JAVAFX.txt
COPYRIGHT include jre LICENSE README.html src.zip THIRDPARTYLICENSEREADME.txt
[[email protected] ~]# cat <<END >>/etc/profile
export JAVA_HOME=/usr/local/java
export PATH=$PATH:/usr/local/java/bin
END
[[email protected]~]# source /etc/profile
[[email protected] ~]# java -version
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)
[[email protected] ~]# tar zxvf apache-tomcat-9.0.10.tar.gz
[[email protected] ~]# mv apache-tomcat-9.0.10 /usr/local/tomcat
[[email protected] ~]# ls /usr/local/tomcat
bin conf lib LICENSE logs NOTICE RELEASE-NOTES RUNNING.txt temp webapps work
[[email protected] ~]# /usr/local/tomcat/bin/startup.sh ##启动apache-tomcat
[[email protected] ~]# netstat -utpln |grep 8080
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 14758/java
安装并配置两台nginx服务器（两台nginx服务器配置相同，在此只列出其中一台配置）
[[email protected] ~]# yum -y install pcre-devel zlib-devel
[[email protected] ~]# useradd -M -s /sbin/nologin nginx
[[email protected] ~]# tar zxvf nginx-1.12.2.tar.gz -C /usr/src/
[[email protected] ~]# cd /usr/src/nginx-1.12.2/
[[email protected] nginx-1.12.2]# ./configure --prefix=/usr/local/nginx --user=nginx --group=nginx --with-http_stub_status_module
[[email protected] nginx-1.12.2]# make && make install
[[email protected] nginx-1.12.2]# cd
[[email protected] ~]# ln -s /usr/local/nginx/sbin/nginx /usr/local/sbin/
[[email protected] ~]# vi /usr/lib/systemd/system/nginx.service
[Unit]
Description=nginxapi
After=network.target
[Service]
Type=forking
PIDFile=/usr/local/nginx/logs/nginx.pid
ExecStart=/usr/local/nginx/sbin/nginx
ExecReload=kill -s HUP $(cat /usr/local/nginx/logs/nginx.pid)
ExecStop=kill -s QUIT $(cat /usr/local/nginx/logs/nginx.pid)
PrivateTmp=Flase
[Install]
WantedBy=multi-user.target
[[email protected] ~]# vi /usr/local/nginx/conf/nginx.conf
34 upstream tomserver {
35 server 192.168.100.105:8080 weight=1;
36 server 192.168.100.106:8080 weight=1;
37 }
50 location ~ \.(asp|aspx|php|jsp|do|js|css|png|jpg)$ {
51 proxy_pass http://tomserver;
52 }
[[email protected] ~]# systemctl start nginx
[[email protected] ~]# systemctl enable nginx
[[email protected] ~]# netstat -utpln |grep nginx
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 3538/nginx: master
安装前端两台负载调度器的keepalived服务与lvs服务（两台调度器配置相同，在此只列出一台配置）
[[email protected] ~]# yum -y install kernel-devel openssl-devel popt-devel
[[email protected] ~]# ls keepalived-1.2.13.tar.gz
keepalived-1.2.13.tar.gz
[[email protected] ~]# tar zxvf keepalived-1.2.13.tar.gz -C /usr/src/
[[email protected] ~]# cd /usr/src/keepalived-1.2.13/
[[email protected] keepalived-1.2.13]# ./configure --prefix=/usr/local/keepalived
[[email protected] keepalived-1.2.13]# make && make install
[[email protected] keepalived-1.2.13]# cd
[[email protected] ~]# mkdir -p /etc/keepalived
[[email protected] ~]# cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
[[email protected] ~]# cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
[[email protected] ~]# cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/
[[email protected] ~]# cp /usr/local/keepalived/sbin/keepalived /usr/sbin/
[[email protected] ~]# chmod 755 /etc/init.d/keepalived
配置master主调度器的keepalived服务并启动
[[email protected] ~]# vi /etc/keepalived/keepalived.conf
global_defs {
router_id HA_TEST_R1 ##本服务器的名称
}
vrrp_instance VI_1 { ##定义VRRP热备实例
state MASTER #MASTER表示主服务器
interface eth0 ##承载VIP地址的物理接口
virtual_router_id 1 ##虚拟路由器的ID号
priority 100 ##优先级，数值越大优先级越高
advert_int 1 ##通告间隔秒数（心跳频率）
authentication { ##认证信息
auth_type PASS ##认证类型
auth_pass 123456 ##密码字串
}
virtual_ipaddress {
192.168.100.95 ##指定漂移地址（VIP）
}
virtual_server 192.168.100.95 80 { ##指定vip地址
delay_loop 5 ##每隔5秒检测一次real server
lb_algo rr
lb_kind DR
protocol TCP
real_server 192.168.100.103 80 { ##指定web集群节点1，在此为nginx1
weight 1
notify_down /etc/keepalived/check.sh ##real server检测失败后执行的脚本
TCP_CHECK {
connect_port 80
connect_timeout 3 ##连接超时
nb_get_retry 3 ##重试连接次数
delay_before_retry 4 ##重试间隔
}
}
real_server 192.168.100.104 80 { ##指定web集群节点2，在此为nginx2
weight 1
notify_down /etc/keepalived/check.sh ##real server检测失败后执行的脚本
TCP_CHECK {
connect_port 80
connect_timeout 3
nb_get_retry 3
delay_before_retry 4
}
}
}
[[email protected] ~]# vi /etc/keepalived/check.sh
#!/bin/bash
echo -e " nginx1(192.168.100.103) or nginx2(192.168.100.104) is down on $(date +%F-%T)" >/root/check_httpd.log
cat /root/check_httpd.log |/usr/local/bin/sendEmail -o message-charset=utf8 -f [email protected] -t [email protected] -s smtp.163.com -u "It's up to it" -xu [email protected] -xp 854365897huhu
:<<END
解释:
-f 表示发送者的邮箱
-t 表示接收者的邮箱
-s 表示SMTP服务器的域名或者ip
-u 表示邮件的主题
-xu 表示SMTP验证的用户名
-xp 表示SMTP验证的密码(注意,这个密码貌似有限制,例如我用d!5neyland就不能被正确识别)
-m 表示邮件的内容
END
:wq
[[email protected] ~]# chmod +x /etc/keepalived/check.sh
[[email protected] ~]# wget http://caspian.dotconf.net/menu/Software/SendEmail/sendEmail-v1.56.tar.gz
[[email protected] ~]# tar zxf sendEmail-v1.56.tar.gz ##安装发送邮件工具
[[email protected] ~]# cd sendEmail-v1.56
[[email protected] sendEmail-v1.56]# mv sendEmail /usr/local/bin/
[[email protected] sendEmail-v1.56]# cd
[[email protected] ~]# cat /etc/fstab |/usr/local/bin/sendEmail -o message-charset=utf8 -f [email protected] -t [email protected] -s smtp.163.com -u "It's up to it" -xu [email protected] -xp ########## ##发送测试邮件，也可用-m指定邮件内容
[[email protected] ~]# modprobe ip_vs ##启动ip_vs模块
[[email protected] ~]# lsmod |grep ip_vs
[[email protected] ~]# echo "modprobe ip_vs" >>/etc/rc.local
[[email protected] ~]# chmod +x /etc/rc.local
[[email protected] ~]# /etc/init.d/keepalived start
Reloading systemd: [ 确定 ]
Starting keepalived (via systemctl): [ 确定 ]
[[email protected] ~]# ip a |grep 192.168.100.95
inet 192.168.100.95/32 scope global eth0
配置backup从调度器的keepalived服务并启动
[[email protected] ~]# vi /etc/keepalived/keepalived.conf
global_defs {
router_id HA_TEST_R2 ##本服务器的名称
}
vrrp_instance VI_1 { ##定义VRRP热备实例
state BACKUP #MASTER表示主服务器
interface eth0 ##承载VIP地址的物理接口
virtual_router_id 1 #虚拟路由器的ID号
priority 99 ##优先级，数值越大优先级越高
advert_int 1 ##通告间隔秒数（心跳频率）
authentication { ##认证信息
auth_type PASS ##认证类型
auth_pass 123456 ##密码字串
}
virtual_ipaddress {
192.168.100.95 ##指定漂移地址（VIP）
}
virtual_server 192.168.100.95 80 { ##指定vip地址
delay_loop 5 ##每隔5秒检测一次real server
lb_algo rr
lb_kind DR
protocol TCP
real_server 192.168.100.103 80 { ##指定web集群节点1，在此为nginx1
weight 1
notify_down /etc/keepalived/check.sh ##real server检测失败后执行的脚本
TCP_CHECK {
connect_port 80
connect_timeout 3 ##连接超时
nb_get_retry 3 ##重试连接次数
delay_before_retry 4 ##重试间隔
}
}
real_server 192.168.100.104 80 { ##指定web集群节点2，在此为nginx2
weight 1
notify_down /etc/keepalived/check.sh ##real server检测失败后执行的脚本
TCP_CHECK {
connect_port 80
connect_timeout 3
nb_get_retry 3
delay_before_retry 4
}
}
}
[[email protected] ~]# vi /etc/keepalived/check.sh
#!/bin/bash
echo -e " nginx1(192.168.100.103) or nginx2(192.168.100.104) is down on $(date +%F-%T)" >/root/check_httpd.log
cat /root/check_httpd.log |/usr/local/bin/sendEmail -o message-charset=utf8 -f [email protected] -t [email protected] -s smtp.163.com -u "It's up to it" -xu [email protected] -xp
:<<END
解释:
-f 表示发送者的邮箱
-t 表示接收者的邮箱
-s 表示SMTP服务器的域名或者ip
-u 表示邮件的主题
-xu 表示SMTP验证的用户名
-xp 表示SMTP验证的密码(注意,这个密码貌似有限制,例如我用d!5neyland就不能被正确识别)
-m 表示邮件的内容
END
:wq
[[email protected] ~]# chmod +x /etc/keepalived/check.sh
[[email protected] ~]# wget http://caspian.dotconf.net/menu/Software/SendEmail/sendEmail-v1.56.tar.gz
[[email protected] ~]# tar zxf sendEmail-v1.56.tar.gz ##安装发送邮件工具
[[email protected] ~]# cd sendEmail-v1.56
[[email protected] sendEmail-v1.56]# mv sendEmail /usr/local/bin/
[[email protected] sendEmail-v1.56]# cd
[[email protected] ~]# modprobe ip_vs ##启动ip_vs模块
[[email protected] ~]# lsmod |grep ip_vs
[[email protected] ~]# echo "modprobe ip_vs" >>/etc/rc.local
[[email protected] ~]# chmod +x /etc/rc.local
[[email protected] ~]# /etc/init.d/keepalived start
Reloading systemd: [ 确定 ]
Starting keepalived (via systemctl): [ 确定 ]
[[email protected] ~]# ip a |grep 192.168.100.95
配置两台nginx在Lvs_DR模式中的网络参数（两台nginx服务器配置相同，在此只列出一台配置）
[[email protected] ~]# cat <<END >/etc/sysconfig/network-scripts/ifcfg-lo:0
DEVICE=lo:0
IPADDR=192.168.100.95
NETMASK=255.255.255.255
ONBOOT=yes
NAME=lo:0
END
[[email protected] ~]# systemctl restart network
[[email protected] ~]# ip a |grep 95
inet 192.168.100.95/32 brd 192.168.100.88 scope global lo:0
客户端测试访问集群, 访问静态网页资源并查看服务器日志
访问动态网站资源并查看服务器日志
安装配置后端存储主机上的mysql服务
[[email protected] ~]# yum -y install mariadb-server mysql
[[email protected] ~]# systemctl start mariadb
[[email protected] ~]# systemctl enable mariadb
Created symlink from /etc/systemd/system/multi-user.target.wants/mariadb.service to /usr/lib/systemd/system/mariadb.service.
[[email protected] ~]# mysqladmin -uroot password ##设置密码为123123
[[email protected] ~]# mysql -uroot -p123123
MariaDB [(none)]> exit
安装配置后端存储主机上的nfs服务，并且将动态项目和静态项目上传并设置nfs共享
[[email protected] ~]# for i in rpcbind nfs;do systemctl enable $i; done
[[email protected] ~]# for i in rpcbind nfs;do systemctl enable $i; done
[[email protected] ~]# mkdir /opt/nginx
[[email protected] ~]# chmod 777 /opt/nginx/
[[email protected] ~]# echo "this is a beautiful page!!!" >>/opt/nginx/index.html ##准备nginx的静态网页资源
[[email protected] ~]# mkdir /opt/tom
[[email protected] ~]# chmod 777 /opt/tom/
[[email protected] ~]# ls /opt/tom/ ##上传超市管理项目的源码
WebRoot
[[email protected] ~]# vi /opt/tom/WebRoot/WEB-INF/classes/database.properties
url=jdbc:mysql://192.168.100.107:3306/smbms?useUnicode=true&characterEncoding=utf-8
user=linuxfan
password=123123
:wq
[[email protected] ~]# vi /etc/exports
/opt/nginx 192.168.100.0/24(rw,sync,no_root_squash)
/opt/tom 192.168.100.0/24(rw,sync,no_root_squash)
[[email protected] ~]# systemctl start rpcbind
[[email protected] ~]# systemctl start nfs
Job for nfs-server.service failed because the control process exited with error code. See "systemctl status nfs-server.service" and "journalctl -xe" for details.
[[email protected] ~]# kill -HUP `cat /run/gssproxy.pid`
[[email protected] ~]# systemctl start nfs
[[email protected] ~]# systemctl enable rpcbind nfs
[[email protected] ~]# showmount -e 192.168.100.107
Export list for 192.168.100.107:
/opt/tom 192.168.100.0/24
/opt/nginx 192.168.100.0/24
两台nginx服务器挂载并读取nfs共享的静态网页资源（两台nginx服务器配置相同，在此只列出一台配置）
[[email protected] ~]# yum -y install nfs-utils rpcbind
[[email protected] ~]# systemctl start rpcbind
[[email protected] ~]# systemctl start nfs
Job for nfs-server.service failed because the control process exited with error code. See "systemctl status nfs-server.service" and "journalctl -xe" for details.
[[email protected] ~]# kill -HUP `cat /run/gssproxy.pid`
[[email protected] ~]# systemctl start nfs
[[email protected] ~]# systemctl enable rpcbind nfs
[[email protected] ~]# showmount -e 192.168.100.107
Export list for 192.168.100.107:
/opt/tom 192.168.100.0/24
/opt/nginx 192.168.100.0/24
[[email protected] ~]# echo "192.168.100.107:/opt/nginx /usr/local/nginx/html/ nfs defaults,_netdev 0 0" >>/etc/fstab
[[email protected] ~]# mount -a
[[email protected] ~]# ls /usr/local/nginx/html/
index.html
[[email protected] ~]# mount |tail -1
192.168.100.107:/opt/nginx on /usr/local/nginx/html type nfs4 (rw,relatime,vers=4.1,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.100.103,local_lock=none,addr=192.168.100.107,_netdev)
两台tomcat服务器挂载并读取nfs共享的动态网站项目（由java编写的超市管理项目），（两台tomcat服务器配置相同，在此只列出其中一台配置）
[[email protected] ~]# yum -y install nfs-utils rpcbind
[[email protected] ~]# systemctl start rpcbind
[[email protected] ~]# systemctl start nfs
Job for nfs-server.service failed because the control process exited with error code. See "systemctl status nfs-server.service" and "journalctl -xe" for details.
[[email protected] ~]# kill -HUP `cat /run/gssproxy.pid`
[[email protected] ~]# systemctl start nfs
[[email protected] ~]# systemctl enable rpcbind nfs
[[email protected] ~]# showmount -e 192.168.100.107
Export list for 192.168.100.107:
/opt/tom 192.168.100.0/24
/opt/nginx 192.168.100.0/24
[[email protected] ~]# echo "192.168.100.107:/opt/tom /usr/local/tomcat/webapps/ nfs defaults,_netdev 0 0" >>/etc/fstab
[[email protected] ~]# mount -a
[[email protected] ~]# ls /usr/local/tomcat/webapps/
WebRoot
[[email protected] ~]# mount |tail -1
192.168.100.107:/opt/tom on /usr/local/tomcat/webapps type nfs4 (rw,relatime,vers=4.1,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.100.105,local_lock=none,addr=192.168.100.107,_netdev)
配置后端mysql数据库
[[email protected] ~]# ls smbms_db.sql
smbms_db.sql
[[email protected] ~]# mysql -uroot -p123123<smbms_db.sql
[[email protected] ~]# mysql -uroot -p123123
MariaDB [(none)]> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| smbms |
| test |
+--------------------+
5 rows in set (0.00 sec)
MariaDB [(none)]> grant all on smbms.* to 'linuxfan'@'192.168.100.%' identified by "123123";
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]> flush privileges;
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]> exit
Bye
客户端访问测试静态网页资源
客户端测试访问动态网站资源（登录后如若访问不了，可以尝试重启tomcat）
将nginx1模拟故障，客户端测试访问以及查看邮件情况
[[email protected] ~]systemctl stop nginx
[[email protected] ~]netstat -utpln |grep 80
[[email protected] ~] cat check_httpd.log
将master主调度器模拟故障，测试客户端访问情况
[[email protected] ~]ip a | grep 95
[[email protected] ~]/etc/init.d/keepalived stop