k8s集群的etcd集群安装配置

环境规划说明

os	wlan ip	lan ip	hostname	roles
centos 7.6	47.74.14.67	172.24.118.168	master01	k8s master;etcd
centos 7.6	47.74.10.221	172.24.118.170	master02	k8s master;etcd
centos 7.6	47.74.3.153	172.24.118.167	node01	k8s node
centos 7.6	47.74.0.203	172.24.118.166	node02	k8s node
centos 7.6	47.74.16.49	172.24.118.169	nginx01	k8s nginx;etcd
centos 7.6	47.74.15.119	172.24.118.171	register01	k8s register

etcd安装配置说明

etcd集群一共三台，后面统一采用https的方式进行调用访问。证书列表：

组件	证书
etcd	ca.pem;server.pem;server-key.pem
flannel	ca.pem;server.pem;server-key.pem
kube-apiserver	ca.pem;server.pem;server-key.pem
kubelet	ca.pem;server.pem;server-key.pem
kube-proxy	ca.pem;server.pem;server-key.pem
kubectl	ca.pem;server.pem;server-key.pem

安装ssl软件

[root@master Deploy]# cat cfssl.sh
curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o /usr/local/bin/cfssl
curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /usr/local/bin/cfssljson
curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /usr/local/bin/cfssl-certinfo
chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson /usr/local/bin/cfssl-certinfo

生成默认配置文件

# mkdir /root/ssl
# cd /root/ssl
# cfssl print-defaults config > ca-config.json



修改配置文件：
#vi ca-config.json
{
    "signing": {
        "default": {
            "expiry": "43800h"
        },
        "profiles": {
            "www": {
                "expiry": "43800h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            },
        }
    }
}

说明：
server auth:client可以采用该ca对server提供证书进行验证
client auth:server可以采用该ca对client提供证书进行验证

配置etcd的证书的签名请求

# cfssl print-defaults csr > ca-csr.json
# vi ca-csr.json
{
    "CN": "etcd CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "hubei",
            "ST": "wuhan"
        }
    ]
}
说明:
size: 必须用2048，用1024通不过
CN: 证书名
C: 国家
L: 省份
ST: 城市

生成证书

[root@master ssl]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca
2019/03/15 21:04:30 [INFO] generating a new CA key and certificate from CSR
2019/03/15 21:04:30 [INFO] generate received request
2019/03/15 21:04:30 [INFO] received CSR
2019/03/15 21:04:30 [INFO] generating key: rsa-2048
2019/03/15 21:04:31 [INFO] encoded CSR
2019/03/15 21:04:31 [INFO] signed certificate with serial number 712960459801665226053583573327268482777791290947
[root@master ssl]# ls ca*
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem

为ETCD生成server证书

[root@master ssl]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
2019/03/15 21:14:59 [INFO] generate received request
2019/03/15 21:14:59 [INFO] received CSR
2019/03/15 21:14:59 [INFO] generating key: rsa-2048
2019/03/15 21:15:00 [INFO] encoded CSR
2019/03/15 21:15:00 [INFO] signed certificate with serial number 111185780868752715604185832446950539067648331791
2019/03/15 21:15:00 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").


[root@master ssl]# ls -lt
total 36
-rw-r--r-- 1 root root 1009 Mar 15 21:15 server.csr
-rw------- 1 root root 1679 Mar 15 21:15 server-key.pem
-rw-r--r-- 1 root root 1326 Mar 15 21:15 server.pem
-rw-r--r-- 1 root root  379 Mar 15 21:14 ca-config.json
-rw-r--r-- 1 root root  301 Mar 15 21:11 server-csr.json
-rw-r--r-- 1 root root  948 Mar 15 21:04 ca.csr
-rw------- 1 root root 1679 Mar 15 21:04 ca-key.pem
-rw-r--r-- 1 root root 1257 Mar 15 21:04 ca.pem
-rw-r--r-- 1 root root  205 Mar 15 21:04 ca-csr.json

etcd下载安装

下载地址：
https://github.com/etcd-io/etcd/releases
关闭selinux
setenforce 0
关闭防火墙
systemctl stop firewalld
查看集群状态的命令

创建工作目录
```
#mkdir -p /opt/etcd/{bin,cfg,ssl}
```

将下载后的文件解压后，将启动命令copy到相关目录

#cp etcdctl /opt/etcd/bin/
#cp etcd /opt/etcd/bin/
#ln -s /opt/etcd/bin/etcd/etcd /usr/local/bin/etcd
#ln -s /opt/etcd/bin/etcdctl /usr/local/bin/etcdctl

创建配置文件（etcd01为例子） ``` #vi /opt/etcd/cfg/etcd #[member] ETCD_NAME="etcd01" ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://172.24.118.168:2380" ETCD_LISTEN_CLIENT_URLS="https://172.24.118.168:2379"

[Cluster]

ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.24.118.168:2380" ETCD_ADVERTISE_CLIENT_URLS="https://172.24.118.168:2379" ETCD_INITIAL_CLUSTER="etcd01=https://172.24.118.168:2380,etcd02=https://172.24.118.170:2380,etcd03=https://172.24.118.171:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new"

- 创建systemd的启动配置

vi /etc/lib/systemd/system/etcd.service

[Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target

[Service] Type=notify EnvironmentFile=/opt/etcd/cfg/etcd ExecStart=/opt/etcd/bin/etcd --name=${ETCD_NAME} --data-dir=${ETCD_DATA_DIR} --listen-peer-urls=${ETCD_LISTEN_PEER_URLS} --listen-client-urls=${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 --advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} --initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} --initial-cluster=${ETCD_INITIAL_CLUSTER} --initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} --initial-cluster-state=new --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --peer-cert-file=/opt/etcd/ssl/server.pem --peer-key-file=/opt/etcd/ssl/server-key.pem --trusted-ca-file=/opt/etcd/ssl/ca.pem --peer-trusted-ca-file=/opt/etcd/ssl/ca.pem Restart=on-failure LimitNOFILE=65535

[Install] WantedBy=multi-user.target

systemctl daemon-reload

systemctl start etcd

注意：第一台启动etcd服务会报错，因为初始化集群时候，会尝试连接其他2台，而其他2台目前没有安装  

安装上述方法安装其他2台etcd server


### 验证etcd集群

[root@master ssl]# etcdctl --ca-file=ca.pem --key-file=server-key.pem --cert-file=server.pem cluster-health member 1743da7b97ff9d08 is healthy: got healthy result from https://172.24.118.171:2379 member 47a967d6ccbf62d5 is healthy: got healthy result from https://172.24.118.170:2379 member 4c3db39630e2ed90 is healthy: got healthy result from https://172.24.118.168:2379 cluster is healthy [root@master ssl]# etcdctl --ca-file=ca.pem --key-file=server-key.pem --cert-file=server.pem --endpoint="https://172.24.118.168:2379" cluster-health member 1743da7b97ff9d08 is healthy: got healthy result from https://172.24.118.171:2379 member 47a967d6ccbf62d5 is healthy: got healthy result from https://172.24.118.170:2379 member 4c3db39630e2ed90 is healthy: got healthy result from https://172.24.118.168:2379 cluster is healthy [root@master ssl]# etcdctl --ca-file=ca.pem --key-file=server-key.pem --cert-file=server.pem --endpoint="https://172.24.118.168:2379,https://172.24.118.170:2379,https://172.24.118.172:2379" cluster-health member 1743da7b97ff9d08 is healthy: got healthy result from https://172.24.118.171:2379 member 47a967d6ccbf62d5 is healthy: got healthy result from https://172.24.118.170:2379 member 4c3db39630e2ed90 is healthy: got healthy result from https://172.24.118.168:2379 cluster is healthy

注意：由于采用了HTTPS认证，需要带上相关的ca和server的证书才能正常访问。否则会证书错误。


## 故障排查
- 证书故障

[root@master ssl]# etcdctl member list client: etcd cluster is unavailable or misconfigured; error #0: x509: certificate signed by unknown authority ; error #1: x509: certificate signed by unknown authority ; error #2: x509: certificate signed by unknown authority

需要带上证书和服务证书访问即可。

- 第一台etcd启动报错

systemctl start etcd

job time out

无需注意，因为第一台需要初始化，链接另外2个节点，另外2个节点服务没有起来。

## etcd常用操作
- member

查看集群组员 [root@master ssl]# etcdctl --ca-file=ca.pem --key-file=server-key.pem --cert-file=server.pem member list 1743da7b97ff9d08: name=etcd03 peerURLs=https://172.24.118.171:2380 clientURLs=https://172.24.118.171:2379 isLeader=false 47a967d6ccbf62d5: name=etcd02 peerURLs=https://172.24.118.170:2380 clientURLs=https://172.24.118.170:2379 isLeader=false 4c3db39630e2ed90: name=etcd01 peerURLs=https://172.24.118.168:2380 clientURLs=https://172.24.118.168:2379 isLeader=true

- cluster-health

查看集群健康状态 [root@master ssl]# etcdctl --ca-file=ca.pem --key-file=server-key.pem --cert-file=server.pem --endpoint="https://172.24.118.168:2379" cluster-health member 1743da7b97ff9d08 is healthy: got healthy result from https://172.24.118.171:2379 member 47a967d6ccbf62d5 is healthy: got healthy result from https://172.24.118.170:2379 member 4c3db39630e2ed90 is healthy: got healthy result from https://172.24.118.168:2379 cluster is healthy

Ryan的sre之旅

不积跬步,无以至千里.不积小流,无以成江海

kubernetes系列文章:etcd的安装

k8s集群的etcd集群安装配置

环境规划说明

etcd安装配置说明

安装ssl软件

生成默认配置文件

配置etcd的证书的签名请求

生成证书

为ETCD生成server证书

etcd下载安装

[Cluster]

vi /etc/lib/systemd/system/etcd.service

systemctl daemon-reload

systemctl start etcd

systemctl start etcd

您还没有登录，请您登录后发表评论。