kubernetes的HPA
环境
| ip | hostname | role | os | kube-version |
|---|---|---|---|---|
| 10.0.7.68 | liran-test-1 | master | centos7 | 1.15 |
| 10.0.7.73 | liran-test-2 | node | centos7 | 1.15 |
| 10.0.7.78 | liran-test-3 | node | centos7 | 1.15 |
原理
HPA(harizontal pod autoscaler):pod水平自动扩容。它用来根据一些监控参数来水平的扩展deployment、rs、rc中的pod。
- 它只支持rs，deployment这种能够伸缩的控制器
- 它目前只支持cpu的参数
- hpa由controller-manager管理的一个控制器
- 默认每隔30s运行一次，由参数horizontal-pod-autoscaler-sync-period定义
- 根据metrics-server中的值和期望值进行对比，来判读执行的动作
- 目前支持resources类型的metric（如，requests和limits的资源），object类型的metric（如，ingress的每秒请求数）和pod类型的metric（如，pod的tps）
算法说明
计算公式
desiredReplicas = ceil[currentReplicas * ( currentMetricValue / desiredMetricValue )]说明
如果现在的监控值为200m，期望的值是100m。那么根据200/100=2,副本集将翻倍。如果监控值是50m，那么副本集50/100=0.5，我们将调过这次扩容。根据参数--horizontal-pod-autoscaler-tolerance来定。默认是1
如果定义了targetAverageValue、targetAverageUtilization。那么currentMetricValue的值将根据所有pod的值进行计算。
扩容时候，未ready的pod和无metrics的pod，按照0计算
缩容时候，未ready的pod不参与计算，无metrics的pod，metrics值按照期望值计算metrics-server
安装
git clone https://github.com/kubernetes-incubator/metrics-server.git修改配置文件,metrics-server-deployment.yml
# cat /root/k8s/metrics-server-0.3.3/deploy/1.8+/metrics-server-deployment.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: metrics-server
namespace: kube-system
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: metrics-server
namespace: kube-system
labels:
k8s-app: metrics-server
spec:
selector:
matchLabels:
k8s-app: metrics-server
template:
metadata:
name: metrics-server
labels:
k8s-app: metrics-server
spec:
serviceAccountName: metrics-server
volumes:
# mount in tmp so we can safely use from-scratch images and/or read-only containers
- name: tmp-dir
emptyDir: {}
containers:
- name: metrics-server
image: harbor-ceshi.stnts.com/k8s/metrics-server-amd64:v0.3.2
imagePullPolicy: IfNotPresent
command:
- /metrics-server
- --kubelet-insecure-tls
- --kubelet-preferred-address-types=InternalDNS,InternalIP,ExternalDNS,ExternalIP,Hostname
- --metric-resolution=30s
volumeMounts:
- name: tmp-dir
mountPath: /tmp执行配置
#kubectl create -f deploy/1.8+/测试
[root@liran-test-1.novalocal 17:11 ~]
# kubectl top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
liran-test-1.novalocal 154m 7% 2300Mi 62%
liran-test-2.novalocal 206m 10% 2052Mi 55%
liran-test-3.novalocal 174m 8% 2119Mi 57%测试
生成nginx的deployment
[root@liran-test-1.novalocal 17:08 ~/calico-policy-test]
# cat nginx-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: ca1-nginx
spec:
replicas: 2
selector:
matchLabels:
name: nginx
template:
metadata:
labels:
name: nginx
spec:
containers:
- name: nginx
image: nginx
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
resources:
requests:
cpu: 100m创建nginx-service
[root@liran-test-1.novalocal 17:08 ~/calico-policy-test]
# cat nginx-svc.yaml
apiVersion: v1
kind: Service
metadata:
name: ca1-nginx-svc
labels:
name: nginx
spec:
ports:
- port: 80
targetPort: 80
protocol: TCP
selector:
name: nginx查看是否支持hpa
[root@liran-test-1.novalocal 17:08 ~/calico-policy-test]
# kubectl get apiservices | grep autoscaling
v1.autoscaling Local True 2d6h
v2beta1.autoscaling Local True 2d6h
v2beta2.autoscaling Local True 2d6h创建hpa
[root@liran-test-1.novalocal 16:58 ~/calico-policy-test]
# kubectl autoscale deployment ca1-nginx --min=2 --max=5 --cpu-percent=10
horizontalpodautoscaler.autoscaling/ca1-nginx autoscaled
或者采用配置文件
# kubectl get hpa ca1-nginx -o yaml --export
Flag --export has been deprecated, This flag is deprecated and will be removed in future.
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
name: ca1-nginx
spec:
maxReplicas: 5
minReplicas: 2
scaleTargetRef:
kind: Deployment
name: ca1-nginx
targetCPUUtilizationPercentage: 10
# kubectl apply -f hpa.yaml创建测试pod
# cat ../test-centos.yaml
apiVersion: v1
kind: Pod
metadata:
name: test-ceph-rbd-3
labels:
app: test-ceph-rbd-3
spec:
containers:
- name: test-ceph-rbd-3
image: centos
command: ["bash", "-c", "sleep 6000"]登录执行命令，提升nginx的cpu
[root@liran-test-1.novalocal 17:09 ~/calico-policy-test]
# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ca1-nginx-svc ClusterIP 172.201.160.60 <none> 80/TCP 47h
[root@liran-test-1.novalocal 17:07 ~]
# kubectl exec -it test-deployment-9c498b79c-d58ph /bin/bash
[root@test-deployment-9c498b79c-d58ph /]# while true;do curl -s 172.201.160.60;done观察
[root@liran-test-1.novalocal 17:01 ~/calico-policy-test]
# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
ca1-nginx Deployment/ca1-nginx 2%/10% 2 5 2 3m3s
[root@liran-test-1.novalocal 17:03 ~/calico-policy-test]
# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
ca1-nginx Deployment/ca1-nginx 19%/10% 2 5 3 4m7s
[root@liran-test-1.novalocal 17:03 ~/calico-policy-test]
# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
ca1-nginx Deployment/ca1-nginx 13%/10% 2 5 4 4m41s
[root@liran-test-1.novalocal 17:03 ~]
# kubectl get pod
NAME READY STATUS RESTARTS AGE
ca1-nginx-f987cfd97-4mx7w 1/1 Running 0 5m35s
ca1-nginx-f987cfd97-pv9jm 1/1 Running 0 50s
ca1-nginx-f987cfd97-qxnml 1/1 Running 0 5m35s
ca1-nginx-f987cfd97-xcxmk 1/1 Running 0 19s
test-deployment-9c498b79c-dr4gq 1/1 Running 0 3m59s
test-deployment-9c498b79c-g5m4p 1/1 Running 0 3m59s删除pod后观察,可能需要几分钟时间
[root@liran-test-1.novalocal 17:10 ~]
# kubectl get pod
NAME READY STATUS RESTARTS AGE
ca1-nginx-f987cfd97-4mx7w 1/1 Running 0 13m
ca1-nginx-f987cfd97-qxnml 1/1 Running 0 13m
test-deployment-9c498b79c-d58ph 1/1 Running 0 7m29s
test-deployment-9c498b79c-g5m4p 1/1 Running 0 11m扩展
https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/
https://kubernetes.io/docs/tasks/federation/administer-federation/hpa/
- 目前只支持基于cpu的扩容，其他的men和自定义的参数还处于v2beta2版本中
- 在生产上面不是很建议采用hpa。如果有峰值，建议提前扩容deployment
- 发版时候会增加cpu的使用，会引起hpa的触发，然后启动的pod会加入到发版的pod统计中。可能会引起版本混乱。