|
个人运维习惯，会专门创建一个app账号，用户部署应用程序。本案例应用程序都部署在/data目录下，将/data权限设置成app
[[email protected] ~]# useradd app
[[email protected] ~]# passwd app
[[email protected] ~]# chown -R app.app /data
前提准备
1）安裝 GCC 编译工具 不然会有编译不过的问题
[[email protected] ~]# yum install -y gcc g++ make gcc-c++ kernel-devel automake autoconf libtool make wget tcl vim ruby rubygems unzip git
2）升级所有的包，防止出现版本过久不兼容问题
[[email protected] ~]# yum -y update
3）关闭防火墙 节点之前需要开放指定端口，为了方便，生产不要禁用
[[email protected] ~]# /etc/init.d/iptables stop
[[email protected] ~]# setenforce 0
[[email protected] ~]# vim /etc/sysconfig/selinux
......
SELINUX=disabled
......
redis cluster集群部署
4）下载并编译安装redis
[[email protected] ~]# su - app
[[email protected] ~]$ mkdir /data/software/
[[email protected] software]$ wget http://download.redis.io/releases/redis-4.0.1.tar.gz
[[email protected] software]$ tar -zvxf redis-4.0.1.tar.gz
[[email protected] software]$ mv redis-4.0.1 /data/
[[email protected] software]$ cd /data/redis-4.0.1/
[[email protected] redis-4.0.1]$ make
--------------------------------------------------------------------------------------
如果因为上次编译失败，有残留的文件，做法如下：
[[email protected] redis-4.0.1]$ make distclean
--------------------------------------------------------------------------------------
5）创建节点
首先在172.16.51.175机器（redis01）上/data/redis-4.0.1目录下创建redis-cluster目录
[[email protected] redis-4.0.1]$ mkdir /data/redis-4.0.1/redis-cluster
接着在redis-cluster目录下，创建名为7000、7001、7002的目录
[[email protected] redis-cluster]$ mkdir 7000
[[email protected] redis-cluster]$ mkdir 7001
[[email protected] redis-cluster]$ mkdir 7002
分别修改这三个配置文件redis.conf
[[email protected] redis-4.0.1]$ cd redis-cluster/
[[email protected] redis-cluster]$ ll
total 12
drwxrwxr-x 2 app app 4096 Nov 16 17:38 7000
drwxrwxr-x 2 app app 4096 Nov 16 17:39 7001
drwxrwxr-x 2 app app 4096 Nov 16 17:39 7002
[[email protected] redis-cluster]$ cat 7000/redis.conf
port 7000
bind 172.16.51.175
daemonize yes
pidfile /var/run/redis_7000.pid
cluster-enabled yes
cluster-config-file nodes_7000.conf
cluster-node-timeout 10100
appendonly yes
[[email protected] redis-cluster]$ cat 7001/redis.conf
port 7001
bind 172.16.51.175
daemonize yes
pidfile /var/run/redis_7001.pid
cluster-enabled yes
cluster-config-file nodes_7001.conf
cluster-node-timeout 10100
appendonly yes
[[email protected] redis-cluster]$ cat 7002/redis.conf
port 7002
bind 172.16.51.175
daemonize yes
pidfile /var/run/redis_7002.pid
cluster-enabled yes
cluster-config-file nodes_7002.conf
cluster-node-timeout 10100
appendonly yes
----------------------------------------------------------------------------------------------------
redis.conf的配置说明：
#端口7000,7001,7002
port 7000
#默认ip为127.0.0.1，需要改为其他节点机器可访问的ip，否则创建集群时无法访问对应的端口，无法创建集群
bind 172.16.51.175
#redis后台运行
daemonize yes
#pidfile文件对应7000，7001，7002
pidfile /var/run/redis_7000.pid
#开启集群，把注释#去掉
cluster-enabled yes
#集群的配置，配置文件首次启动自动生成 7000，7001，7002
cluster-config-file nodes_7000.conf
#请求超时，默认15秒，可自行设置
cluster-node-timeout 10100
#aof日志开启，有需要就开启，它会每次写操作都记录一条日志
appendonly yes
----------------------------------------------------------------------------------------------------
接着在另外两台机器上(172.16.51.176，172.16.51.178)重复以上三步，只是把目录改为7003、7004、7005和7006、7007、7008，对应的配置文件也按照这个规则修改即可（即修改redis.conf文件中的端口就行了）
6）启动集群（依次启动7000-7008端口）
#第一个节点机器上执行 3个节点
[[email protected] redis-cluster]$ for((i=0;i<=2;i++)); do /data/redis-4.0.1/src/redis-server /data/redis-4.0.1/redis-cluster/700$i/redis.conf; done
#第二个节点机器上执行 3个节点
[[email protected] redis-cluster]$ for((i=3;i<=5;i++)); do /data/redis-4.0.1/src/redis-server /data/redis-4.0.1/redis-cluster/700$i/redis.conf; done
#第三个节点机器上执行 3个节点
[[email protected] redis-cluster]$ for((i=6;i<=8;i++)); do /data/redis-4.0.1/src/redis-server /data/redis-4.0.1/redis-cluster/700$i/redis.conf; done
7）检查服务
检查各 Redis 各个节点启动情况
[[email protected] redis-cluster]$ ps -ef | grep redis
app 2564 2405 0 20:13 pts/0 00:00:00 grep redis
app 15197 1 0 17:57 ? 00:00:05 /data/redis-4.0.1/src/redis-server 172.16.51.175:7000 [cluster]
app 15199 1 0 17:57 ? 00:00:05 /data/redis-4.0.1/src/redis-server 172.16.51.175:7001 [cluster]
app 15201 1 0 17:57 ? 00:00:05 /data/redis-4.0.1/src/redis-server 172.16.51.175:7002 [cluster]
[[email protected] redis-cluster]$ ps -ef | grep redis
app 2566 2405 0 20:13 pts/0 00:00:00 grep redis
app 15197 1 0 17:57 ? 00:00:05 /data/redis-4.0.1/src/redis-server 172.16.51.175:7000 [cluster]
app 15199 1 0 17:57 ? 00:00:05 /data/redis-4.0.1/src/redis-server 172.16.51.175:7001 [cluster]
app 15201 1 0 17:57 ? 00:00:05 /data/redis-4.0.1/src/redis-server 172.16.51.175:7002 [cluster]
8）安装 Ruby（需要切换到root账号下进行安装，app账号下权限不够）
[[email protected] ~]# yum -y install ruby ruby-devel rubygems rpm-build
[[email protected] ~]# gem install redis
-----------------------------------------------------------------------------------------------------
注意：在centos6.x下执行上面的"gem install redis"操作可能会报错，坑很多！
默认yum安装的ruby版本是1.8.7，版本太低，需要升级到ruby2.2以上，否则执行上面安装会报错！
首先安装rvm（或者直接下载证书：https://pan.baidu.com/s/1slTyJ7n **：7uan 下载并解压后直接执行"curl -L get.rvm.io | bash -s stable"即可）
[[email protected] ~]# curl -L get.rvm.io | bash -s stable //可能会报错，需要安装提示进行下面一步操作
[[email protected] ~]# curl -sSL https://rvm.io/mpapis.asc | gpg2 --import - //然后再接着执行：curl -L get.rvm.io | bash -s stable
[[email protected] ~]# find / -name rvm.sh
/etc/profile.d/rvm.sh
[[email protected] ~]# source /etc/profile.d/rvm.sh
[[email protected] ~]# rvm requirements
然后升级ruby到2.3
[[email protected] ~]# rvm install ruby 2.3.1
[[email protected] ~]# ruby -v
ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_64-linux]
列出所有ruby版本
[[email protected] ~]# rvm list
设置默认的版本
[[email protected] ~]# rvm --default use 2.3.1
更新下载源
[[email protected] ~]# gem sources --add https://gems.ruby-china.org/ --remove https://rubygems.org
https://gems.ruby-china.org/ added to sources
source https://rubygems.org not present in cache
[[email protected] ~]# gem sources
*** CURRENT SOURCES ***
https://rubygems.org/
https://gems.ruby-china.org/
最后就能顺利安装了
[[email protected] src]# gem install redis
Successfully installed redis-4.0.1
Parsing documentation for redis-4.0.1
Done installing documentation for redis after 1 seconds
1 gem installed
-----------------------------------------------------------------------------------------------------
9）创建集群
千万注意：在任意一台上运行即可，不要在每台机器上都运行，一台就够了！！！！
Redis 官方提供了 redis-trib.rb 这个工具，就在解压目录的 src 目录中
[[email protected] ~]# su - app
[[email protected] ~]$ /data/redis-4.0.1/src/redis-trib.rb create --replicas 1 172.16.51.175:7000 172.16.51.175:7001 172.16.51.175:7002 172.16.51.176:7003 172.16.51.176:7004 172.16.51.176:7005 172.16.51.178:7006 172.16.51.178:7007 172.16.51.178:7008
出现下面信息，从下面信息可以看出，本案例三台服务器启动9个实例，配置成4主5从，其中有一个是一主两从，其他3个都是一主一从。
>>> Creating cluster
>>> Performing hash slots allocation on 9 nodes...
Using 4 masters:
172.16.51.175:7000
172.16.51.176:7003
172.16.51.178:7006
172.16.51.175:7001
Adding replica 172.16.51.176:7004 to 172.16.51.175:7000
Adding replica 172.16.51.178:7007 to 172.16.51.176:7003
Adding replica 172.16.51.175:7002 to 172.16.51.178:7006
Adding replica 172.16.51.176:7005 to 172.16.51.175:7001
Adding replica 172.16.51.178:7008 to 172.16.51.175:7000
M: 7c622ac191edd40dd61d9b79b27f6f69d02a5bbf 172.16.51.175:7000
slots:0-4095 (4096 slots) master
M: 44c81c15b01d992cb9ede4ad35477ec853d70723 172.16.51.175:7001
slots:12288-16383 (4096 slots) master
S: 38f03c27af39723e1828eb62d1775c4b6e2c3638 172.16.51.175:7002
replicates f1abb62a8c9b448ea14db421bdfe3f1d8075189c
M: 987965baf505a9aa43e50e46c76189c51a8f17ec 172.16.51.176:7003
slots:4096-8191 (4096 slots) master
S: 6555292fed9c5d52fcf5b983c441aff6f96923d5 172.16.51.176:7004
replicates 7c622ac191edd40dd61d9b79b27f6f69d02a5bbf
S: 2b5ba254a0405d4efde4c459867b15176f79244a 172.16.51.176:7005
replicates 44c81c15b01d992cb9ede4ad35477ec853d70723
M: f1abb62a8c9b448ea14db421bdfe3f1d8075189c 172.16.51.178:7006
slots:8192-12287 (4096 slots) master
S: eb4067373d36d8a8df07951f92794e67a6aac022 172.16.51.178:7007
replicates 987965baf505a9aa43e50e46c76189c51a8f17ec
S: 2919e041dd3d1daf176d6800dcd262f4e727f366 172.16.51.178:7008
replicates 7c622ac191edd40dd61d9b79b27f6f69d02a5bbf
Can I set the above configuration? (type 'yes' to accept): yes
输入 yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join.........
>>> Performing Cluster Check (using node 172.16.51.175:7000)
M: 7c622ac191edd40dd61d9b79b27f6f69d02a5bbf 172.16.51.175:7000
slots:0-4095 (4096 slots) master
2 additional replica(s)
S: 6555292fed9c5d52fcf5b983c441aff6f96923d5 172.16.51.176:7004
slots: (0 slots) slave
replicates 7c622ac191edd40dd61d9b79b27f6f69d02a5bbf
M: 44c81c15b01d992cb9ede4ad35477ec853d70723 172.16.51.175:7001
slots:12288-16383 (4096 slots) master
1 additional replica(s)
S: 2919e041dd3d1daf176d6800dcd262f4e727f366 172.16.51.178:7008
slots: (0 slots) slave
replicates 7c622ac191edd40dd61d9b79b27f6f69d02a5bbf
M: f1abb62a8c9b448ea14db421bdfe3f1d8075189c 172.16.51.178:7006
slots:8192-12287 (4096 slots) master
1 additional replica(s)
S: eb4067373d36d8a8df07951f92794e67a6aac022 172.16.51.178:7007
slots: (0 slots) slave
replicates 987965baf505a9aa43e50e46c76189c51a8f17ec
S: 38f03c27af39723e1828eb62d1775c4b6e2c3638 172.16.51.175:7002
slots: (0 slots) slave
replicates f1abb62a8c9b448ea14db421bdfe3f1d8075189c
S: 2b5ba254a0405d4efde4c459867b15176f79244a 172.16.51.176:7005
slots: (0 slots) slave
replicates 44c81c15b01d992cb9ede4ad35477ec853d70723
M: 987965baf505a9aa43e50e46c76189c51a8f17ec 172.16.51.176:7003
slots:4096-8191 (4096 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
10）关闭集群
推荐做法：
[[email protected] ~]$ pkill redis
[[email protected] ~]$ pkill redis
[[email protected] ~]$ pkill redis
或者循环节点逐个关闭
[[email protected] ~]$ for((i=0;i<=2;i++)); do /opt/redis-4.0.1/src/redis-cli -c -h 172.16.51.175 -p 700$i shutdown; done
[[email protected] ~]$ for((i=3;i<=5;i++)); do /opt/redis-4.0.1/src/redis-cli -c -h 172.16.51.176 -p 700$i shutdown; done
[[email protected] ~]$ for((i=6;i<=8;i++)); do /opt/redis-4.0.1/src/redis-cli -c -h 172.16.51.178 -p 700$i shutdown; done
11）集群验证
连接集群测试
参数-C可连接到集群，因为redis.conf将bind改为了ip地址，所以-h参数不可以省略，-p参数为端口号
可以先在172.16.51.175机器redis 7000 的节点set一个key
[[email protected] ~]$ /data/redis-4.0.1/src/redis-cli -h 172.16.51.175 -c -p 7000
172.16.51.175:7000> set name www.ymq.io
-> Redirected to slot [5798] located at 172.16.51.176:7003
OK
172.16.51.176:7003> get name
"www.ymq.io"
172.16.51.176:7003>
由上面信息可发现redis set name 之后重定向到172.16.51.176机器 redis 7003 这个节点
然后在172.16.51.178机器redis 7008 的节点get一个key
[[email protected] ~]$ /data/redis-4.0.1/src/redis-cli -h 172.16.51.178 -c -p 7008
172.16.51.178:7008> get name
-> Redirected to slot [5798] located at 172.16.51.176:7003
"www.ymq.io"
172.16.51.176:7003>
发现redis get name 重定向到172.16.51.176机器 redis 7003 这个节点.
如果看到这样的现象，说明redis cluster集群已经是可用的了!!!!!!
12）检查集群状态(通过下面的命令，可以看到本案例实现的是4主5从，4个主节点会默认分配到三个机器上，每个机器上都要有master；另：创建集群的时候可以指定master和slave。这里我是默认创建的)
[[email protected] ~]$ /data/redis-4.0.1/src/redis-cli -h 172.16.51.175 -c -p 7000
172.16.51.175:7000>
[[email protected] ~]$ /data/redis-4.0.1/src/redis-trib.rb check 172.16.51.175:7000
>>> Performing Cluster Check (using node 172.16.51.175:7000)
M: 5a43e668f53ff64da68be31afe6dc6ea1f3c14c5 172.16.51.175:7000
slots:0-4095 (4096 slots) master
2 additional replica(s)
M: c64b0839e0199f73c5c192cc8c90f12c999f79b2 172.16.51.175:7001
slots:12288-16383 (4096 slots) master
1 additional replica(s)
S: 81347f01cf38d8f0faef1ad02676ebb4cffbec9e 172.16.51.176:7005
slots: (0 slots) slave
replicates c64b0839e0199f73c5c192cc8c90f12c999f79b2
M: da5dde3f2f02c232784bf3163f5f584b8cf046f2 172.16.51.178:7006
slots:8192-12287 (4096 slots) master
1 additional replica(s)
M: b217ab2a6c05497af3b2a859c1bb6b3fae5e0d92 172.16.51.176:7003
slots:4096-8191 (4096 slots) master
1 additional replica(s)
S: 0420c49fbc9f1fe16066d189265cca2f5e71c86e 172.16.51.178:7007
slots: (0 slots) slave
replicates b217ab2a6c05497af3b2a859c1bb6b3fae5e0d92
S: 5ad89453fb36e50ecc4560de6b4acce1dbbb78b3 172.16.51.176:7004
slots: (0 slots) slave
replicates 5a43e668f53ff64da68be31afe6dc6ea1f3c14c5
S: bbd1f279b99b95cf00ecbfab22b6b8dd5eb05989 172.16.51.178:7008
slots: (0 slots) slave
replicates 5a43e668f53ff64da68be31afe6dc6ea1f3c14c5
S: e95407b83bfeb30e3cc537161eadc372d6aa1fa2 172.16.51.175:7002
slots: (0 slots) slave
replicates da5dde3f2f02c232784bf3163f5f584b8cf046f2
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
13）列出集群节点
列出集群当前已知的所有节点（node），以及这些节点的相关信息
[[email protected] ~]$ /data/redis-4.0.1/src/redis-cli -h 172.16.51.175 -c -p 7000
172.16.51.175:7000> cluster nodes
5a43e668f53ff64da68be31afe6dc6ea1f3c14c5 172.16.51.175:[email protected] myself,master - 0 1510836027000 1 connected 0-4095
c64b0839e0199f73c5c192cc8c90f12c999f79b2 172.16.51.175:[email protected] master - 0 1510836030068 2 connected 12288-16383
81347f01cf38d8f0faef1ad02676ebb4cffbec9e 172.16.51.176:[email protected] slave c64b0839e0199f73c5c192cc8c90f12c999f79b2 0 1510836031000 6 connected
da5dde3f2f02c232784bf3163f5f584b8cf046f2 172.16.51.178:[email protected] master - 0 1510836031000 7 connected 8192-12287
b217ab2a6c05497af3b2a859c1bb6b3fae5e0d92 172.16.51.176:[email protected] master - 0 1510836030000 4 connected 4096-8191
0420c49fbc9f1fe16066d189265cca2f5e71c86e 172.16.51.178:[email protected] slave b217ab2a6c05497af3b2a859c1bb6b3fae5e0d92 0 1510836029067 8 connected
5ad89453fb36e50ecc4560de6b4acce1dbbb78b3 172.16.51.176:[email protected] slave 5a43e668f53ff64da68be31afe6dc6ea1f3c14c5 0 1510836032672 5 connected
bbd1f279b99b95cf00ecbfab22b6b8dd5eb05989 172.16.51.178:[email protected] slave 5a43e668f53ff64da68be31afe6dc6ea1f3c14c5 0 1510836031000 9 connected
e95407b83bfeb30e3cc537161eadc372d6aa1fa2 172.16.51.175:[email protected] slave da5dde3f2f02c232784bf3163f5f584b8cf046f2 0 1510836031672 7 connected
14）打印集群信息
[[email protected] ~]$ /data/redis-4.0.1/src/redis-cli -h 172.16.51.175 -c -p 7000
172.16.51.175:7000> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:9
cluster_size:4
cluster_current_epoch:9
cluster_my_epoch:1
cluster_stats_messages_ping_sent:8627
cluster_stats_messages_pong_sent:8581
cluster_stats_messages_sent:17208
cluster_stats_messages_ping_received:8573
cluster_stats_messages_pong_received:8626
cluster_stats_messages_meet_received:8
cluster_stats_messages_received:17207
------------------------------------------------------------------------------------------------
[[email protected] src]# pwd
/data/redis-4.0.1/src
[[email protected] src]# ./redis-trib.rb help
Usage: redis-trib <command> <options> <arguments ...>
create host1:port1 ... hostN:portN
--replicas <arg>
check host:port
info host:port
fix host:port
--timeout <arg>
reshard host:port
--from <arg>
--to <arg>
--slots <arg>
--yes
--timeout <arg>
--pipeline <arg>
rebalance host:port
--weight <arg>
--auto-weights
--use-empty-masters
--timeout <arg>
--simulate
--pipeline <arg>
--threshold <arg>
add-node new_host:new_port existing_host:existing_port
--slave
--master-id <arg>
del-node host:port node_id
set-timeout host:port milliseconds
call host:port command arg arg .. arg
import host:port
--from <arg>
--copy
--replace
help (show this help)
For check, fix, reshard, del-node, set-timeout you can specify the host and port of any working node in the cluster.
上面已经多次出现了slot这个词，略为解释一下：
redis-cluster把整个集群的存储空间划分为16384个slot(译为：插槽？)，当9个实例分为3主6从时，相当于整个cluster中有3组HA的节点，
3个master会平均分摊所有slot，每次向cluster中的key做操作时（比如：读取/写入缓存），redis会对key值做CRC32算法处理，得到一个数值，
然后再对16384取模，通过余数判断该缓存项应该落在哪个slot上，确定了slot，也就确定了保存在哪个master节点上，当cluster扩容或删除
节点时，只需要将slot重新分配即可（即：把部分slot从一些节点移动到其它节点）。
---------------------------------------------------------------------------------------------------------
在代码里连接以上redis cluster集群节点配置如下：
spring.redis.cluster.nodes = 172.16.51.175:7000,172.16.51.175:7001,172.16.51.175:7002,172.16.51.176:7003,172.16.51.176:7004,172.16.51.176:7005,172.16.51.178:7006,172.16.51.178:7007,172.16.51.178:7008
|
