etcd集群的备份与还原
环境
| ip | hostname | role |
|---|---|---|
| 10.0.7.68 | liran-test-1 | etcd-test-1 |
| 10.0.7.73 | liran-test-2 | etcd-test-2 |
| 10.0.7.78 | liran-test-3 | etcd-test-3 |
etcd配置
ETCD_OPT="--name liran-test-1 --initial-advertise-peer-urls http://10.0.7.68:2380 --listen-peer-urls http://10.0.7.68:2380 --listen-client-urls http://10.0.7.68:2379,http://127.0.0.1:2379 --advertise-client-urls http://10.0.7.68:2379 --initial-cluster-token liran-etcd-1 --initial-cluster liran-test-1=http://10.0.7.68:2380,liran-test-2=http://10.0.7.73:2380,liran-test-3=http://10.0.7.78:2380 --initial-cluster-state new"备份kubernetes
[root@liran-test-1.novalocal 10:02 /opt/etcd]
# du -sh /var/lib/kubelet/
136K /var/lib/kubelet/
[root@liran-test-1.novalocal 10:03 /opt/etcd]
# cp -rf /var/lib/kubelet/ /backup/kubelet/备份etcd
[root@liran-test-1.novalocal 10:01 /opt/etcd]
# export ETCDCTL_API=3
[root@liran-test-1.novalocal 10:01 /opt/etcd]
# etcdctl --endpoints localhost:2379 snapshot save snapshot.db
Snapshot saved at snapshot.db还原
注意：以下操作都在export ETCDCTL_API=3下面进行
节点1
etcdctl snapshot restore snapshot.db --data-dir=data --name=liran-test-1 --endpoints=10.0.7.68:2379 --initial-advertise-peer-urls=http://10.0.7.68:2380 --initial-cluster-token=liran-etcd-1 --initial-cluster=liran-test-1=http://10.0.7.68:2380,liran-test-2=http://10.0.7.73:2380,liran-test-3=http://10.0.7.78:2380节点2
[root@liran-test-2.novalocal 10:21 /opt/etcd]
# etcdctl snapshot restore snapshot.db --data-dir=data --name=liran-test-2 --endpoints=10.0.7.73:2379 --initial-advertise-peer-urls=http://10.0.7.73:2380 --initial-cluster-token=liran-etcd-1 --initial-cluster=liran-test-1=http://10.0.7.68:2380,liran-test-2=http://10.0.7.73:2380,liran-test-3=http://10.0.7.78:2380
2019-06-28 10:26:07.008087 I | mvcc: restore compact to 182158
2019-06-28 10:26:07.025597 I | etcdserver/membership: added member 804dfdf475146e06 [http://10.0.7.68:2380] to cluster 582905de1d321827
2019-06-28 10:26:07.025716 I | etcdserver/membership: added member b376717912ab1128 [http://10.0.7.78:2380] to cluster 582905de1d321827
2019-06-28 10:26:07.025758 I | etcdserver/membership: added member b7145e476cf92607 [http://10.0.7.73:2380] to cluster 582905de1d321827节点3
[root@liran-test-3.novalocal 10:26 /opt/etcd]
# ./bin/etcdctl snapshot restore snapshot.db --data-dir=data --name=liran-test-3 --endpoints=10.0.7.78:2379 --initial-advertise-peer-urls=http://10.0.7.78:2380 --initial-cluster-token=liran-etcd-1 --initial-cluster=liran-test-1=http://10.0.7.68:2380,liran-test-2=http://10.0.7.73:2380,liran-test-3=http://10.0.7.78:2380
2019-06-28 10:27:11.819956 I | mvcc: restore compact to 182158
2019-06-28 10:27:11.842832 I | etcdserver/membership: added member 804dfdf475146e06 [http://10.0.7.68:2380] to cluster 582905de1d321827
2019-06-28 10:27:11.842894 I | etcdserver/membership: added member b376717912ab1128 [http://10.0.7.78:2380] to cluster 582905de1d321827
2019-06-28 10:27:11.842927 I | etcdserver/membership: added member b7145e476cf92607 [http://10.0.7.73:2380] to cluster 582905de1d321827查看member
[root@liran-test-1.novalocal 10:38 /opt/etcd]
# etcdctl member list
804dfdf475146e06, started, liran-test-1, http://10.0.7.68:2380, http://10.0.7.68:2379
b376717912ab1128, started, liran-test-3, http://10.0.7.78:2380, http://10.0.7.78:2379
b7145e476cf92607, started, liran-test-2, http://10.0.7.73:2380, http://10.0.7.73:2379查看key
[root@liran-test-1.novalocal 10:39 /opt/etcd]
# etcdctl get / --prefix --keys-only | more
/registry/apiextensions.k8s.io/customresourcedefinitions/bgpconfigurations.crd.projectcalico.
org
/registry/apiextensions.k8s.io/customresourcedefinitions/bgppeers.crd.projectcalico.org
/registry/apiextensions.k8s.io/customresourcedefinitions/blockaffinities.crd.projectcalico.or
g
/registry/apiextensions.k8s.io/customresourcedefinitions/clusterinformations.crd.projectcalic
o.org
/registry/apiextensions.k8s.io/customresourcedefinitions/felixconfigurations.crd.projectcalic
o.org查看集群健康，在版本2下面进行
[root@liran-test-3.novalocal 10:29 /opt/etcd]
# export ETCDCTL_API=2
[root@liran-test-3.novalocal 10:43 /opt/etcd]
# etcdctl cluster-health
member 804dfdf475146e06 is healthy: got healthy result from http://10.0.7.68:2379
member b376717912ab1128 is healthy: got healthy result from http://10.0.7.78:2379
member b7145e476cf92607 is healthy: got healthy result from http://10.0.7.73:2379
cluster is healthy重启apiserver
systemctl restart apiserver查看pod
[root@liran-test-1.novalocal 10:41 /opt/etcd]
# kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-6fb584dd97-q6x2g 1/1 Running 1 19h
kube-system calico-node-ldwqr 1/1 Running 0 18h
kube-system calico-node-ts4vt 1/1 Running 0 19h
kube-system calico-node-vdr24 1/1 Running 0 19h