k8s中的pvc扩容(ceph-rbd)
前言
在k8s的1.11版本中，k8s默认集成了pv的扩容接口。通过编辑pvc来达到扩容的目的。需要开启feature gate和ExpandPersistentVolumes， PresistentVolumeClaimResize。在1.11版本之后这个特性默认开启。
目前支持的volume插件: AWS-EBS,GCE-PD,AZURE DISK, Azure File, Clusterfs, Cinder, Portworx, Ceph RBD。管理员通过allowVolumeExpansion字段值为true来开启volume扩展功能
# cat rbd-class.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: ceph-rbd
provisioner: ceph.com/rbd
#provisioner: kubernetes.io/rbd
parameters:
monitors: 10.0.7.52:6789
pool: kube
adminId: admin
adminSecretNamespace: kube-system
adminSecretName: ceph-admin-secret
userId: kube
userSecretNamespace: kube-system
userSecretName: ceph-secret
fsType: ext4
imageFormat: "2"
imageFeatures: layering
allowVolumeExpansion: true
reclaimPolicy: Retain目前扩容的文件系统为XFS, EXT3, EXT4
开启功能
采用kubeadm安装的话，默认feture-gate都是开启的。如果是二进制安装的话，需要手动开启。
apiserver
KUBE_ADMISSION_CONTROL="--admission-control=PersistentVolumeClaimResize,Initializers,
NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,
NodeRestriction,ResourceQuota"
--feature-gates=ExpandPersistentVolumes=truecontroller
--feature-gates=ExpandPersistentVolumes=true官方文档测试(未成功)
创建pvc
# cat pvc-test.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: test-pvc
spec:
accessModes:
- ReadWriteOnce
storageClassName: ceph-rbd
resources:
requests:
storage: 2Gi创建pod
pod创建
# cat pod-test.yaml | grep -v "^$"
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: ceph-rbd-pvc-test
spec:
replicas: 1
template:
metadata:
labels:
app: ceph-rbd-pvc-test
spec:
containers:
- name: ceph-rbd-pvc-test
image: busybox:latest
command:
- sleep
- "360000"
volumeMounts:
- name: ceph-rbd-test
mountPath: /data/ceph-rbd
readOnly: false
volumes:
- name: ceph-rbd-test
persistentVolumeClaim:
claimName: test-pvc在pod中实验
# kubectl get pod
NAME READY STATUS RESTARTS AGE
ceph-rbd-pvc-test-59d4869cfb-g9kb8 1/1 Running 0 70s
rbd-provisioner-77cc657d45-l6hfw 1/1 Running 95 55d
[root@k8s-ceshi-01.novalocal 17:48 ~/k8s/ceph/rbd/deploy/rbac/test]
# kubectl exec -it ceph-rbd-pvc-test-59d4869cfb-g9kb8 /bin/sh
/ # df -hT
Filesystem Type Size Used Available Use% Mounted on
overlay overlay 40.0G 30.8G 9.2G 77% /
tmpfs tmpfs 64.0M 0 64.0M 0% /dev
tmpfs tmpfs 5.8G 0 5.8G 0% /sys/fs/cgroup
/dev/vda1 xfs 40.0G 30.8G 9.2G 77% /dev/termination-log
/dev/rbd0 ext4 2.9G 9.0M 2.9G 0% /data/ceph-rbd
/dev/vda1 xfs 40.0G 30.8G 9.2G 77% /etc/resolv.conf
/dev/vda1 xfs 40.0G 30.8G 9.2G 77% /etc/hostname
/dev/vda1 xfs 40.0G 30.8G 9.2G 77% /etc/hosts
shm tmpfs 64.0M 0 64.0M 0% /dev/shm
tmpfs tmpfs 5.8G 12.0K 5.8G 0% /var/run/secrets/kubernetes.io/serviceaccount
tmpfs tmpfs 5.8G 0 5.8G 0% /proc/acpi
tmpfs tmpfs 64.0M 0 64.0M 0% /proc/kcore
tmpfs tmpfs 64.0M 0 64.0M 0% /proc/keys
tmpfs tmpfs 64.0M 0 64.0M 0% /proc/timer_list
tmpfs tmpfs 64.0M 0 64.0M 0% /proc/timer_stats
tmpfs tmpfs 64.0M 0 64.0M 0% /proc/sched_debug
tmpfs tmpfs 5.8G 0 5.8G 0% /proc/scsi
tmpfs tmpfs 5.8G 0 5.8G 0% /sys/firmware
/ # cd /data/ceph-rbd/
/data/ceph-rbd # ls
lost+found
/data/ceph-rbd # vi abc
/data/ceph-rbd # echo "1234" >> 123
/data/ceph-rbd # ls -lt
total 24
-rw-r--r-- 1 root root 5 Sep 18 09:49 123
-rw-r--r-- 1 root root 13 Sep 18 09:49 abc
drwx------ 2 root root 16384 Sep 18 09:47 lost+foundpvc扩容
修改pvc的容量
# cat pvc-test.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: test-pvc
spec:
accessModes:
- ReadWriteOnce
storageClassName: ceph-rbd
resources:
requests:
storage: 5Gi查看pvc状态,可以看到状态在resizing状态
# kubectl get pvc test-pvc -o yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"test-pvc","namespace":"default"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"5Gi"}},"storageClassName":"ceph-rbd"}}
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
volume.beta.kubernetes.io/storage-provisioner: ceph.com/rbd
creationTimestamp: "2019-09-18T09:47:22Z"
finalizers:
- kubernetes.io/pvc-protection
name: test-pvc
namespace: default
resourceVersion: "43700029"
selfLink: /api/v1/namespaces/default/persistentvolumeclaims/test-pvc
uid: 502a6637-d9f9-11e9-a3ea-fa163e8116b5
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
storageClassName: ceph-rbd
volumeMode: Filesystem
volumeName: pvc-502a6637-d9f9-11e9-a3ea-fa163e8116b5
status:
accessModes:
- ReadWriteOnce
capacity:
storage: 3Gi
conditions:
- lastProbeTime: null
lastTransitionTime: "2019-09-18T09:51:14Z"
status: "True"
type: Resizing
phase: Bound过一会儿等待文件扩容完成
# kubectl get pvc test-pvc -o yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"test-pvc","namespace":"default"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"50Gi"}},"storageClassName":"ceph-rbd"}}
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
volume.beta.kubernetes.io/storage-provisioner: ceph.com/rbd
creationTimestamp: "2019-09-23T03:57:09Z"
finalizers:
- kubernetes.io/pvc-protection
name: test-pvc
namespace: default
resourceVersion: "44746226"
selfLink: /api/v1/namespaces/default/persistentvolumeclaims/test-pvc
uid: 37512c5c-ddb6-11e9-a3ea-fa163e8116b5
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
storageClassName: ceph-rbd
volumeMode: Filesystem
volumeName: pvc-37512c5c-ddb6-11e9-a3ea-fa163e8116b5
status:
accessModes:
- ReadWriteOnce
capacity:
storage: 3Gi
conditions:
- lastProbeTime: null
lastTransitionTime: "2019-09-23T08:38:44Z"
message: Waiting for user to (re-)start a pod to finish file system resize of
volume on node.
status: "True"
type: FileSystemResizePending
phase: Bound然后重启pod既可以达到扩容pod的目的。
在线扩容
需要开启ExpandInUsePersistentVolumes=true。同样支持GCE-PD, AWS-EBS, Cinder, Ceph RBD。目前还处于测试阶段。该方案扩容后不需要重启pod。
通过rbd命令扩容
该方式直接通过rbd命令直接操作pod的rbd镜像，来达到扩容的目的。
- 获取pv的镜像
# kubectl get pv pvc-d4aeea1a-ddde-11e9-a3ea-fa163e8116b5 -o yaml | grep image image: kubernetes-dynamic-pvc-d4bc72fc-ddde-11e9-86f1-d2ff46c62203 - 查看pv的rbd信息 ```
[root@k8s-ceshi-01.novalocal 16:46 ~/k8s/ceph/rbd/deploy/rbac]
rbd list -m 10.0.7.52 --key=AQBRRgNduPgTIxAAp9KLXRxjFYLLXwiENdGvFw== --pool kube kubernetes-dynamic-pvc-50444df4-d9f9-11e9-94f6-12f1cfd47f5e
kubernetes-dynamic-pvc-d4bc72fc-ddde-11e9-86f1-d2ff46c62203
[root@k8s-ceshi-01.novalocal 16:54 ~/k8s/ceph/rbd/deploy/rbac]
rbd info kubernetes-dynamic-pvc-d4bc72fc-ddde-11e9-86f1-d2ff46c62203 -m 10.0.7.52 --key=AQBRRgNduPgTIxAAp9KLXRxjFYLLXwiENdGvFw== --pool kube
rbd image 'kubernetes-dynamic-pvc-d4bc72fc-ddde-11e9-86f1-d2ff46c62203': size 3 GiB in 768 objects order 22 (4 MiB objects) id: acb36b8b4567 block_name_prefix: rbd_data.acb36b8b4567 format: 2 features: layering op_features: flags: create_timestamp: Mon Sep 23 16:47:53 2019
- 扩容rbd[root@k8s-ceshi-01.novalocal 16:25 ~/k8s/ceph/rbd/deploy/rbac]
rbd resize --image kubernetes-dynamic-pvc-376331bd-ddb6-11e9-86f1-d2ff46c62203 --size 51200 -m 10.0.7.52 --key=AQBRRgNduPgTIxAAp9KLXRxjFYLLXwiENdGvFw== --pool kube
Resizing image: 100% complete...done.
- 在挂载节点上面扩容[root@k8s-ceshi-01.novalocal 16:54 ~/k8s/ceph/rbd/deploy/rbac/test]
kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
ceph-rbd-pvc-test-59d4869cfb-kpl25 1/1 Running 0 7m39s 172.88.18.87 k8s-ceshi-05.novalocal
[root@k8s-ceshi-05.novalocal 16:30 /var/lib/kubelet]
df -hT | grep kubernetes-dynamic-pvc-d4bc72fc-ddde-11e9-86f1-d2ff46c62203
/dev/rbd0 ext4 2.9G 9.0M 2.9G 1% /var/lib/kubelet/plugins/kubernetes.io/rbd/mounts/kube-image-kubernetes-dynamic-pvc-d4bc72fc-ddde-11e9-86f1-d2ff46c62203
[root@k8s-ceshi-05.novalocal 16:30 /var/lib/kubelet]
blockdev --getsize64 /dev/rbd0
53687091200
[root@k8s-ceshi-05.novalocal 16:30 /var/lib/kubelet]
resize2fs /dev/rbd0
resize2fs 1.42.9 (28-Dec-2013) Filesystem at /dev/rbd0 is mounted on /var/lib/kubelet/plugins/kubernetes.io/rbd/mounts/kube-image-kubernetes-dynamic-pvc-376331bd-ddb6-11e9-86f1-d2ff46c62203; on-line resizing required old_desc_blocks = 5, new_desc_blocks = 7 The filesystem on /dev/rbd0 is now 13107200 blocks long.
如果是xfs的话，采用xfs_growfs -d /mnt来进行扩容
- 查看pod状态[root@k8s-ceshi-01.novalocal 17:01 ~/k8s/ceph/rbd/deploy/rbac/test]
kubectl exec -it ceph-rbd-pvc-test-59d4869cfb-kpl25 /bin/sh
/ # df -hT Filesystem Type Size Used Available Use% Mounted on overlay overlay 40.0G 31.0G 9.0G 78% / tmpfs tmpfs 64.0M 0 64.0M 0% /dev tmpfs tmpfs 5.8G 0 5.8G 0% /sys/fs/cgroup /dev/rbd0 ext4 4.9G 12.0M 4.8G 0% /data/ceph-rbd
- 修改pv
修改pv的状态为5G[root@k8s-ceshi-01.novalocal 17:06 ~/k8s/ceph/rbd/deploy/rbac/test]
kubectl get pv pvc-d4aeea1a-ddde-11e9-a3ea-fa163e8116b5 -o yaml
apiVersion: v1 kind: PersistentVolume metadata: annotations: pv.kubernetes.io/provisioned-by: ceph.com/rbd rbdProvisionerIdentity: ceph.com/rbd creationTimestamp: "2019-09-23T08:47:53Z" finalizers:
- kubernetes.io/pv-protection name: pvc-d4aeea1a-ddde-11e9-a3ea-fa163e8116b5 resourceVersion: "44749887" selfLink: /api/v1/persistentvolumes/pvc-d4aeea1a-ddde-11e9-a3ea-fa163e8116b5 uid: d4d41995-ddde-11e9-a3ea-fa163e8116b5 spec: accessModes:
- ReadWriteOnce
capacity:
storage: 5Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: test-pvc
namespace: default
resourceVersion: "44747590"
uid: d4aeea1a-ddde-11e9-a3ea-fa163e8116b5
persistentVolumeReclaimPolicy: Retain
rbd:
fsType: ext4
image: kubernetes-dynamic-pvc-d4bc72fc-ddde-11e9-86f1-d2ff46c62203
keyring: /etc/ceph/keyring
monitors:
- 10.0.7.52:6789 pool: kube secretRef: name: ceph-secret namespace: kube-system user: kube storageClassName: ceph-rbd volumeMode: Filesystem status: phase: Bound ```
- 查看pvc状态 ``` # kubectl get pvc test-pvc -o yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"test-pvc","namespace":"default"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"50Gi"}},"storageClassName":"ceph-rbd"}} pv.kubernetes.io/bind-completed: "yes" pv.kubernetes.io/bound-by-controller: "yes" volume.beta.kubernetes.io/storage-provisioner: ceph.com/rbd creationTimestamp: "2019-09-23T03:57:09Z" finalizers:
- kubernetes.io/pvc-protection name: test-pvc namespace: default resourceVersion: "44746226" selfLink: /api/v1/namespaces/default/persistentvolumeclaims/test-pvc uid: 37512c5c-ddb6-11e9-a3ea-fa163e8116b5 spec: accessModes:
- ReadWriteOnce resources: requests: storage: 5Gi storageClassName: ceph-rbd volumeMode: Filesystem volumeName: pvc-37512c5c-ddb6-11e9-a3ea-fa163e8116b5 status: accessModes:
- ReadWriteOnce capacity: storage: 3Gi conditions:
- lastProbeTime: null
lastTransitionTime: "2019-09-23T08:38:44Z"
message: Waiting for user to (re-)start a pod to finish file system resize of
volume on node.
status: "True"
type: FileSystemResizePending
phase: Bound
# kubectl delete pod ceph-rbd-pvc-test-59d4869cfb-cfgzw pod "ceph-rbd-pvc-test-59d4869cfb-cfgzw" deleted- 重启pod
# kubectl get pvc,pv NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/test-pvc Bound pvc-37512c5c-ddb6-11e9-a3ea-fa163e8116b5 50Gi RWO ceph-rbd 4h45m- 查看状态
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE persistentvolume/pvc-37512c5c-ddb6-11e9-a3ea-fa163e8116b5 50Gi RWO Retain Bound default/test-pvc ceph-rbd 4h45m
## 扩展
cbd命令说明
- 查看info信息rbd info kubernetes-dynamic-pvc-376331bd-ddb6-11e9-86f1-d2ff46c62203 -m 10.0.7.52 --key=AQBRRgNduPgTIxAAp9KLXRxjFYLLXwiENdGvFw== --pool kube --format=json
{"name":"kubernetes-dynamic-pvc-376331bd-ddb6-11e9-86f1-d2ff46c62203","id":"ac726b8b4567","size":3221225472,"objects":768,"order":22,"object_size":4194304,"block_name_prefix":"rbd_data.ac726b8b4567","format":2,"features":["layering"],"op_features":[],"flags":[],"create_timestamp":"Mon Sep 23 11:57:12 2019"}
- 删除image[root@k8s-ceshi-01.novalocal 16:46 ~/k8s/ceph/rbd/deploy/rbac]
rbd remove kubernetes-dynamic-pvc-376331bd-ddb6-11e9-86f1-d2ff46c62203 -m 10.0.7.52 --key=AQBRRgNduPgTIxAAp9KLXRxjFYLLXwiENdGvFw== --pool kube
Removing image: 100% complete...done.
- 列出imagesrbd list -m 10.0.7.52 --key=AQBRRgNduPgTIxAAp9KLXRxjFYLLXwiENdGvFw== --pool kube kubernetes-dynamic-pvc-50444df4-d9f9-11e9-94f6-12f1cfd47f5e
- 扩容imagesrbd resize --image kubernetes-dynamic-pvc-376331bd-ddb6-11e9-86f1-d2ff46c62203 --size 51200 -m 10.0.7.52 --key=AQBRRgNduPgTIxAAp9KLXRxjFYLLXwiENdGvFw== --pool kube
Resizing image: 100% complete...done.
## 排错
注意：该错误排查，仅仅应用于官方文档的扩容方式
### rbd文件找不到
报错Warning VolumeResizeFailed 3m (x75 over 4h) volume_expand Error expanding volume "default/resize" of plugin kubernetes.io/rbd : rbd info failed, error: executable file not found in $PATH
分析：
在controller中找不到rbd这个命令。
解决办法：
我们要在controller中安装rbd-common这个包。如果fs的话，安装ceph-fs-common
### 执行输出报错E0923 07:36:36.319172 1 nestedpendingoperations.go:267] Operation for "\"37512c5c-ddb6-11e9-a3ea-fa163e8116b5\"" failed. No retries permitted until 2019-09-23 07:38:38.319000883 +0000 UTC m=+156611.810101860 (durationBeforeRetry 2m2s). Error: "error expanding volume \"default/test-pvc\" of plugin \"kubernetes.io/rbd\": rbd info failed, error: parse rbd info output failed: 2019-09-23 07:36:36.239778 7f913ad24100 -1 did not load config file, using default settings.\n{\"name\":\"kubernetes-dynamic-pvc-376331bd-ddb6-11e9-86f1-d2ff46c62203\",\"size\":3221225472,\"objects\":768,\"order\":22,\"object_size\":4194304,\"block_name_prefix\":\"rbd_data.ac726b8b4567\",\"format\":2,\"features\":[\"layering\"],\"flags\":[]}, invalid character '-' after top-level value"
分析：
rbd中解析输出的时候报错。
正常情况[root@k8s-ceshi-01.novalocal 15:10 ~/k8s/ceph/rbd/deploy/rbac]
rbd info kubernetes-dynamic-pvc-376331bd-ddb6-11e9-86f1-d2ff46c62203 -m 10.0.7.52 --key=AQBRRgNduPgTIxAAp9KLXRxjFYLLXwiENdGvFw== --pool kube --format=json
{"name":"kubernetes-dynamic-pvc-376331bd-ddb6-11e9-86f1-d2ff46c62203","id":"ac726b8b4567","size":3221225472,"objects":768,"order":22,"object_size":4194304,"block_name_prefix":"rbd_data.ac726b8b4567","format":2,"features":["layering"],"op_features":[],"flags":[],"create_timestamp":"Mon Sep 23 11:57:12 2019"}
而异常的情况[root@k8s-ceshi-01.novalocal 15:10 ~/k8s/ceph/rbd/deploy/rbac]
rbd info kubernetes-dynamic-pvc-376331bd-ddb6-11e9-86f1-d2ff46c62203 -m 10.0.7.52 --key=AQBRRgNduPgTIxAAp9KLXRxjFYLLXwiENdGvFw== --pool kube --format=json
did not load config file, using default settings. 2019-09-23 15:43:47.500 7f6d1861e840 -1 Errors while parsing config file! 2019-09-23 15:43:47.500 7f6d1861e840 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory 2019-09-23 15:43:47.500 7f6d1861e840 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory 2019-09-23 15:43:47.500 7f6d1861e840 -1 parse_file: cannot open ceph.conf: (2) No such file or directory 2019-09-23 15:43:47.501 7f6d1861e840 -1 Errors while parsing config file! 2019-09-23 15:43:47.501 7f6d1861e840 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory 2019-09-23 15:43:47.501 7f6d1861e840 -1 parse_file: cannot open /root/.ceph/ceph.conf: (2) No such file or directory 2019-09-23 15:43:47.501 7f6d1861e840 -1 parse_file: cannot open ceph.conf: (2) No such file or directory {"name":"kubernetes-dynamic-pvc-376331bd-ddb6-11e9-86f1-d2ff46c62203","id":"ac726b8b4567","size":3221225472,"objects":768,"order":22,"object_size":4194304,"block_name_prefix":"rbd_data.ac726b8b4567","format":2,"features":["layering"],"op_features":[],"flags":[],"create_timestamp":"Mon Sep 23 11:57:12 2019"} ``` 然后在解析的时候报错了。导致rbd的info命令执行失败。