在AWS EC2上搭建ETCD集群

一、以http方式搭建etcd集群

1、准备好三台机器,三台机器如下：

IP地址主机名称安装服务

172.31.72.142 master1 Etcd、Maser节点

172.31.82.187 master2 Etcd、Node节点

172.31.11.86 master3 Etcd、Node节点

在172.31.72.142机器，执行hostnamectl set-hostname master1

在172.31.82.187 机器，执行hostnamectl set-hostname master2

在172.31.82.187 机器，执行hostnamectl set-hostname master3

在三台机器，vim /etc/hosts 添加master1,master2,master3

[[email protected] ssl]# cat /etc/hosts

172.31.72.142 master1

172.31.82.187 master2

172.31.11.86 master3

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4

::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

2、防火墙打开端口2380，2379。

在3台主机都执行以下命令：

# cat /etc/sysconfig/iptables

Generated by iptables-save v1.4.21 on Thu Jul 5 07:05:39 2018

*filter

:INPUT ACCEPT [0:0]

:FORWARD ACCEPT [0:0]

:OUTPUT ACCEPT [1654:201815]

-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT

-A INPUT -p icmp -j ACCEPT

-A INPUT -i lo -j ACCEPT

-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT

-A INPUT -p tcp -m state --state NEW -m tcp --dport 21 -j ACCEPT

-A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT

-A INPUT -p tcp -m state --state NEW -m tcp --dport 2379 -j ACCEPT

-A INPUT -p tcp -m state --state NEW -m tcp --dport 2380 -j ACCEPT

-A INPUT -j REJECT --reject-with icmp-host-prohibited

-A FORWARD -j REJECT --reject-with icmp-host-prohibited

COMMIT

# systemctl reload iptables

# systemctl restart iptables

工作中，由于etcd都是内网访问，所以都是关闭防火墙，关闭iptables，利用AWS安全组，控制对外访问端口。

停掉iptables :

systemctl stop iptables

systemctl disable iptables

3、在三台机器上，分别安装etcd，分别执行yum install -y etcd。我这里安装的版本etcd-3.2.18

4、centos7安装etcd的，默认配置文件路径。/etc/etcd/etcd.conf。在三台主机分别执行 vim /etc/etcd/etcd.conf，配置文件修改点如下：

master1(72.31.72.142):

#[Member]

ETCD_DATA_DIR=“/app/etcd"

ETCD_LISTEN_PEER_URLS="http://172.31.72.142:2380,http://127.0.0.1:2380”

ETCD_LISTEN_CLIENT_URLS="http://172.31.72.142:2379,http://127.0.0.1:2379”

ETCD_NAME="etcd1"

#[Clustering]

ETCD_INITIAL_ADVERTISE_PEER_URLS="http://172.31.72.142:2380"

ETCD_ADVERTISE_CLIENT_URLS="http://172.31.72.142:2379”

ETCD_INITIAL_CLUSTER="etcd1=http://172.31.72.142:2380,etcd2=http://172.31.82.187:2380,etcd3=http://172.31.11.86:2380"

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-token"

ETCD_INITIAL_CLUSTER_STATE="new"

master2(172.31.82.187):

#[Member]

ETCD_DATA_DIR=“/app/etcd”

ETCD_LISTEN_PEER_URLS="http://172.31.82.187:2380,http://127.0.0.1:2380”

ETCD_LISTEN_CLIENT_URLS="http://172.31.82.187:2379,http://127.0.0.1:2379”

ETCD_NAME="etcd2"

#[Clustering]

ETCD_INITIAL_ADVERTISE_PEER_URLS="http://172.31.82.187:2380"

ETCD_ADVERTISE_CLIENT_URLS="http://172.31.82.187:2379”

ETCD_INITIAL_CLUSTER="etcd1=http://172.31.72.142:2380,etcd2=http://172.31.82.187:2380,etcd3=http://172.31.11.86:2380"

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-token"

ETCD_INITIAL_CLUSTER_STATE="new”

master3(172.31.11.86 ):

#[Member]

ETCD_DATA_DIR=“/app/etcd”

ETCD_LISTEN_PEER_URLS="http://172.31.11.86:2380,http://127.0.0.1:2380”

ETCD_LISTEN_CLIENT_URLS="http://172.31.11.86:2379,http://127.0.0.1:2379”

ETCD_NAME="etcd3"

#[Clustering]

ETCD_INITIAL_ADVERTISE_PEER_URLS="http://172.31.11.86:2380"

ETCD_ADVERTISE_CLIENT_URLS="http://172.31.11.86:2379”

ETCD_INITIAL_CLUSTER="etcd1=http://172.31.72.142:2380,etcd2=http://172.31.82.187:2380,etcd3=http://172.31.11.86:2380"

ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster-token"

ETCD_INITIAL_CLUSTER_STATE="new”

5、由于yum安装的etcd默认启动，是以etcd用户启动，可能会出现ETCD_DATA_DIR目录权限问题，这里，直接修改三台机器，都以root身份启动，

vim /lib/systemd/system/etcd.service

[Service]

User=etcd

将User=etcd修改为User=root

6、重新加载配置文件。

三台机器执行以下命令：

systemctl daemon-reload

systemctl enable etcd

systemctl start etcd

7、验证一下etcd集群

在随便一台机器上执行以下命令

ETCDCTL_API=3 etcdctl member list

如果出现，三台机器列表，代表集群(http方式）搭建成功。

二、搭建SSL etcd集群

在master1机器执行以下步骤：

1、下载cfssl工具，并按照cfssl

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64

wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64

chmod +x cfssl*

mv cfssl_linux-amd64 /usr/local/bin/cfssl

mv cfssljson_linux-amd64 /usr/local/bin/cfssljson

2、设置/usr/local/bin可以直接加载到用户环境中，这里我用的AWS，需要设置一下才能把/usr/local/bin路径加载到用户环境。

在/etc/profile文件末尾，添加如下：

PATH="$PATH:/usr/local/bin"

export PATH

添加完成，执行source /etc/profile 重载当前用户环境

3、生成证书

#cd /etc/etcd

#mkdir ssl

#cd ssl

#touch build-server.sh

#vim build-server.sh

echo '{"CN":"CA","key":{"algo":"rsa","size":2048}}' | cfssl gencert -initca - | cfssljson -bare ca -

echo '{"signing":{"default":{"expiry":"43800h","usages":["signing","key encipherment","server auth","client auth"]}}}' > ca-config.json

export ADDRESS=172.31.72.142,172.31.82.187,172.31.11.86,master1,master2,master3

export NAME=kubernetes

echo '{"CN":"'$NAME'","hosts":["localhost","127.0.0.1","0.0.0.0","172.31.72.142","172.31.82.187","172.31.11.86","master1","master2","master3"],"key":{"algo":"rsa","size":2048}}' | cfssl gencert -config=ca-config.json -ca=ca.pem -ca-key=ca-key.pem -hostname="$ADDRESS" - | cfssljson -bare $NAME

export ADDRESS=

export NAME=client

#chmod +x build-server.sh

#sh build-server.sh

在AWS EC2上搭建ETCD集群

4、将ssl目录复制到master2，master3机器上。我这里用的AWS,执行以下命令，上传ssl目录到master1，master2机器。

scp -i /home/centos/aws.pem -rp /etc/etcd/ssl [email protected]:/home/centos

5、配置etcd启动证书

master1(172.31.72.142)

[[email protected] ssl]# vim /lib/systemd/system/etcd.service

[Unit]

Description=Etcd Server

After=network.target

After=network-online.target

Wants=network-online.target

[Service]

Type=notify

WorkingDirectory=/var/lib/etcd/

EnvironmentFile=-/etc/etcd/etcd.conf

User=root

# set GOMAXPROCS to number of processors

ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd --name=etcd1 \

--data-dir=/data/etcd \

--listen-client-urls https://172.31.72.142:2379,https://127.0.0.1:2379 \

--advertise-client-urls https://172.31.72.142:2379,https://127.0.0.1:2379 \

--listen-peer-urls https://172.31.72.142:2380 \

--initial-advertise-peer-urls https://172.31.72.142:2380 \

--initial-cluster etcd1=https://172.31.72.142:2380,etcd2=https://172.31.82.187:2380,etcd3=https://172.31.11.86:2380 \

--initial-cluster-token etcd-cluster-token \

--initial-cluster-state new \

--cert-file=/etc/etcd/ssl/kubernetes.pem \

--key-file=/etc/etcd/ssl/kubernetes-key.pem \

--peer-cert-file=/etc/etcd/ssl/kubernetes.pem \

--peer-key-file=/etc/etcd/ssl/kubernetes-key.pem \

--trusted-ca-file=/etc/etcd/ssl/ca.pem \

--peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \

--peer-client-cert-auth=true \

--client-cert-auth=true"

Restart=on-failure

LimitNOFILE=65536

[Install]

WantedBy=multi-user.target

master2(172.31.82.187)

[[email protected] etcd] cd /etc/etcd

[[email protected] etcd] cp -rf /home/centos/ssl .

[[email protected] etcd] vim /lib/systemd/system/etcd.service

[Unit]

Description=Etcd Server

After=network.target

After=network-online.target

Wants=network-online.target

[Service]

Type=notify

WorkingDirectory=/var/lib/etcd/

EnvironmentFile=-/etc/etcd/etcd.conf

User=root

# set GOMAXPROCS to number of processors

ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd --name=etcd2 \

--data-dir=/data/etcd \

--listen-client-urls https://172.31.82.187:2379,https://127.0.0.1:2379 \

--advertise-client-urls https://172.31.82.187:2379,https://127.0.0.1:2379 \

--listen-peer-urls https://172.31.82.187:2380 \

--initial-advertise-peer-urls https://172.31.82.187:2380 \

--initial-cluster etcd1=https://172.31.72.142:2380,etcd2=https://172.31.82.187:2380,etcd3=https://172.31.11.86:2380 \

--initial-cluster-token etcd-cluster-token \

--initial-cluster-state new \

--cert-file=/etc/etcd/ssl/kubernetes.pem \

--key-file=/etc/etcd/ssl/kubernetes-key.pem \

--peer-cert-file=/etc/etcd/ssl/kubernetes.pem \

--peer-key-file=/etc/etcd/ssl/kubernetes-key.pem \

--trusted-ca-file=/etc/etcd/ssl/ca.pem \

--peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \

--peer-client-cert-auth=true \

--client-cert-auth=true"

Restart=on-failure

LimitNOFILE=65536

[Install]

WantedBy=multi-user.target

master3(172.31.11.86)

[[email protected] etcd] cd /etc/etcd

[[email protected] etcd] cp -rf /home/centos/ssl .

[[email protected] etcd] vim /lib/systemd/system/etcd.service

[Unit]

Description=Etcd Server

After=network.target

After=network-online.target

Wants=network-online.target

[Service]

Type=notify

WorkingDirectory=/var/lib/etcd/

EnvironmentFile=-/etc/etcd/etcd.conf

User=root

# set GOMAXPROCS to number of processors

ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd --name=etcd3 \

--data-dir=/data/etcd \

--listen-client-urls https://172.31.11.86:2379,https://127.0.0.1:2379 \

--advertise-client-urls https://172.31.11.86:2379,https://127.0.0.1:2379 \

--listen-peer-urls https://172.31.11.86:2380 \

--initial-advertise-peer-urls https://172.31.11.86:2380 \

--initial-cluster etcd1=https://172.31.72.142:2380,etcd2=https://172.31.82.187:2380,etcd3=https://172.31.11.86:2380 \

--initial-cluster-token etcd-cluster-token \

--initial-cluster-state new \

--cert-file=/etc/etcd/ssl/kubernetes.pem \

--key-file=/etc/etcd/ssl/kubernetes-key.pem \

--peer-cert-file=/etc/etcd/ssl/kubernetes.pem \

--peer-key-file=/etc/etcd/ssl/kubernetes-key.pem \

--trusted-ca-file=/etc/etcd/ssl/ca.pem \

--peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \

--peer-client-cert-auth=true \

--client-cert-auth=true"

Restart=on-failure

LimitNOFILE=65536

[Install]

WantedBy=multi-user.target

6、在master1，master2，master3执行，systemctl daemon-reload。

如果按照之前http的方式搭建etcd集群，这次重新启动etcd，如果出现，cluster id不匹配的话，请执行以下命令：

# systemctl stop etcd

# rm -rf /etc/data

# systemctl start etcd

7、测试SSL etcd集群安装情况，在任何一台机器测试，如下所示：

[[email protected] ssl]# ETCDCTL_API=3 etcdctl --write-out=table \

--cert=/etc/etcd/ssl/client.pem \

--key=/etc/etcd/ssl/client-key.pem \

--cacert=/etc/etcd/ssl/ca.pem \

--endpoints=https://master1:2379,https://master2:2379,https://master3:2379 \

member list

+------------------+---------+-------+----------------------------+---------------------------------------------------+

+------------------+---------+-------+----------------------------+---------------------------------------------------+

+------------------+---------+-------+----------------------------+---------------------------------------------------+

[[email protected] ssl]#

在AWS EC2上搭建ETCD集群

相关推荐