主题
实验三:高级调度策略、拓扑均衡、优先级抢占与容灾驱逐实战
0. 实验目标与环境基线
本实验利用你的 Mac mini M4 + Lima 三节点集群(lima-master、lima-node01、lima-node02),通过给节点打上不同的硬件标签与可用区(topology.kubernetes.io/zone)标签,完整演练 Kubernetes 调度器的预选与优选、节点亲和性、Pod 间亲和与反亲和、污点容忍、Even Pod Spreading (maxSkew) 跨拓扑均匀分布、PDB 破坏预算防并发驱逐中断、PriorityClass 高优先级抢占,以及使用 limactl stop 模拟节点掉线与 Descheduler 重平衡。
包含的 7 个核心实操点
- 实验点 3.1:节点标签管理、
nodeSelector与NodeAffinity(硬亲和requiredvs 软亲和preferred权重打分) - 实验点 3.2:
PodAffinity(同节点聚集减少网络跳数)与PodAntiAffinity(跨节点反亲和打散容灾) - 实验点 3.3:节点污点(
NoSchedule/PreferNoSchedule/NoExecute)与 PodTolerations(含tolerationSeconds限时驱逐) - 实验点 3.4:Even Pod Spreading(
topologySpreadConstraints+maxSkew: 1)实现跨节点/跨可用区严格均匀分布 - 实验点 3.5:
PriorityClass高优先级 Pod 抢占(Preemption)低优先级 Pod 算力实验 - 实验点 3.6:配置
PDB (PodDisruptionBudget)拦截kubectl drain防止维护期服务全灭 - 实验点 3.7:节点失联故障演练(
limactl stop node02)与 Descheduler 重调度器自动恢复平衡验证
1. 实验前准备:为 lima-node01 和 lima-node02 规划拓扑域与硬件标签
在 lima-master 上执行:
bash
limactl shell master
kubectl create namespace lab3-schedule
kubectl config set-context --current --namespace=lab3-schedule
# 为 lima-node01 和 lima-node02 打上模拟的可用区 (zone) 标签和磁盘/算力标签
kubectl label node lima-node01 topology.kubernetes.io/zone=zone-a disktype=nvme accelerator=gpu --overwrite
kubectl label node lima-node02 topology.kubernetes.io/zone=zone-b disktype=sata --overwrite
# 查看节点标签
kubectl get nodes -L topology.kubernetes.io/zone,disktype,accelerator2. 实验点 3.1:nodeSelector 与 NodeAffinity 软硬亲和验证
原理回顾:
requiredDuringSchedulingIgnoredDuringExecution(硬亲和):必须满足,否则 Pod 卡在Pending。preferredDuringSchedulingIgnoredDuringExecution(软亲和):优选阶段给匹配节点加weight分,若不满足也能退而求其次调度到其他节点。
bash
# 1. 部署一个要求硬亲和 zone-a/zone-b,同时软亲和优先选择 disktype=sata (即 lima-node02) 的 Pod
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: affinity-demo
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: topology.kubernetes.io/zone
operator: In
values: ["zone-a", "zone-b"]
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
preference:
matchExpressions:
- key: disktype
operator: In
values: ["sata"]
containers:
- name: pause
image: registry.k8s.io/pause:3.10
EOF
kubectl wait --for=condition=Ready pod/affinity-demo --timeout=60s
# 验证它因软亲和 100 分加分而精准落在了 lima-node02 上!
kubectl get pod affinity-demo -o wide3. 实验点 3.2:PodAffinity 聚集与 PodAntiAffinity 反亲和打散
原理回顾:
- 聚集(PodAffinity):将 Web 后端与本地 Redis 缓存调度到同一个
kubernetes.io/hostname拓扑域,降低跨节点网络开销。 - 打散(PodAntiAffinity):强制同一个 Deployment 的 2 个副本不得落在同一台物理机上。
bash
# 1. 部署 2 副本 Redis,使用硬反亲和确保 node01 和 node02 各跑 1 个
cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: redis-cache
spec:
replicas: 2
selector:
matchLabels:
app: redis-cache
template:
metadata:
labels:
app: redis-cache
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
app: redis-cache
topologyKey: kubernetes.io/hostname
containers:
- name: redis
image: busybox:1.36
command: ["sleep", "3600"]
EOF
kubectl rollout status deployment/redis-cache
# 验证 2 个 Pod 严格分散在 lima-node01 和 lima-node02
kubectl get pods -l app=redis-cache -o wide4. 实验点 3.3:节点污点(NoSchedule vs NoExecute)与限时容忍驱逐
原理回顾:
NoSchedule:新 Pod 不进驻,但节点上已运行的 Pod 不受影响。NoExecute:不仅新 Pod 不进驻,节点上已运行的未容忍 Pod 会被立即驱逐(若配置了tolerationSeconds: 10,则容忍 10 秒后被驱逐)。
bash
# 1. 在 lima-node02 上创建一个带 NoExecute 10秒限时容忍的 Pod
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: taint-victim
spec:
nodeName: lima-node02
tolerations:
- key: "maintenance"
operator: "Equal"
value: "true"
effect: "NoExecute"
tolerationSeconds: 10
containers:
- name: app
image: busybox:1.36
command: ["sleep", "3600"]
EOF
kubectl wait --for=condition=Ready pod/taint-victim --timeout=60s
# 2. 先给 lima-node02 打上 NoSchedule 污点,验证 taint-victim 依然稳稳运行
kubectl taint nodes lima-node02 maintenance=true:NoSchedule
kubectl get pod taint-victim -o wide
# 3. 升级污点为 NoExecute,观察 taint-victim 撑过 10 秒容忍期后被自动驱逐销毁!
kubectl taint nodes lima-node02 maintenance=true:NoExecute --overwrite
echo "Waiting 12 seconds for tolerationSeconds to expire..."
sleep 12
kubectl get pod taint-victim
# 4. 实验完毕立即清除 lima-node02 上的测试污点(末尾加减号 -)
kubectl taint nodes lima-node02 maintenance:NoSchedule-
kubectl taint nodes lima-node02 maintenance:NoExecute-5. 实验点 3.4:Even Pod Spreading (topologySpreadConstraints) 均匀分布
原理回顾:之前我们学到 PodAntiAffinity 硬反亲和只允许每个节点最多 1 个 Pod(当副本数为 4、节点数为 2 时,第 3 和第 4 个 Pod 会卡在 Pending)。而使用 topologySpreadConstraints 并设置 maxSkew: 1,可以让 4 个副本在 zone-a(lima-node01)和 zone-b(lima-node02)上实现完美的 2 : 2 均匀分布!
bash
cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: even-spread-app
spec:
replicas: 4
selector:
matchLabels:
app: even-spread
template:
metadata:
labels:
app: even-spread
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: even-spread
containers:
- name: app
image: busybox:1.36
command: ["sleep", "3600"]
EOF
kubectl rollout status deployment/even-spread-app
# 验证 lima-node01 (zone-a) 恰好 2 个,lima-node02 (zone-b) 恰好 2 个!
kubectl get pods -l app=even-spread -o wide
# 清理该 Deployment 释放算力供下一节抢占实验使用
kubectl delete deployment redis-cache even-spread-app6. 实验点 3.5:PriorityClass 高优先级抢占(Preemption)实战
原理回顾:你的 lima-node01 和 lima-node02 各有 2 vCPU(即 2000m)。当低优先级 Pod 把两个节点的 CPU requests 账面额度占满后,此时提交一个高优先级(PriorityClass)Pod,Scheduler 会自动触发抢占(Preemption),驱逐低优先级 Pod 为核心业务腾出空间!
bash
# 1. 创建高低两个 PriorityClass
cat <<'EOF' | kubectl apply -f -
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: high-priority-core
value: 100000
globalDefault: false
description: "Core payment service priority"
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: low-priority-batch
value: 100
globalDefault: false
description: "Offline batch task priority"
EOF
# 2. 部署 2 个低优先级 Pod,每个申请 1200m CPU(由于每台 Worker 只有 2000m CPU,每台 Worker 只能放 1 个,从而占满两台 Worker)
cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: low-prio-filler
spec:
replicas: 2
selector:
matchLabels:
app: filler
template:
metadata:
labels:
app: filler
spec:
priorityClassName: low-priority-batch
containers:
- name: pause
image: busybox:1.36
command: ["sleep", "3600"]
resources:
requests:
cpu: "1200m"
memory: "128Mi"
EOF
kubectl rollout status deployment/low-prio-filler
kubectl get pods -l app=filler -o wide
# 3. 现在提交一个申请 1200m CPU 的高优先级 Pod,观察它抢占并踢走其中一个低优先级 Pod!
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: high-prio-vip
spec:
priorityClassName: high-priority-core
containers:
- name: vip
image: busybox:1.36
command: ["sleep", "3600"]
resources:
requests:
cpu: "1200m"
memory: "128Mi"
EOF
sleep 8
# 观察 high-prio-vip 成功 Running,而其中一个 low-prio-filler 被抢占后变成 Pending!
kubectl get pods -o wide
kubectl get events --sort-by=.lastTimestamp | tail -n 10
# 清理抢占实验资源
kubectl delete pod high-prio-vip
kubectl delete deployment low-prio-filler
kubectl delete priorityclass high-priority-core low-priority-batch7. 实验点 3.6:PDB (PodDisruptionBudget) 拦截节点排空(kubectl drain)保护
原理回顾:当执行 kubectl drain 维护节点时,如果业务设置了 PodDisruptionBudget (minAvailable: 2),而当前总共只有 2 个副本在线,API Server 的 Eviction 接口会阻止同时把两个副本都杀掉,必须保持至少 2 个可用。
bash
# 1. 创建 2 副本业务,并配置 PDB 要求 minAvailable: 2(即不允许任何自愿中断导致可用数低于 2)
cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: pdb-protected-app
spec:
replicas: 2
selector:
matchLabels:
app: pdb-app
template:
metadata:
labels:
app: pdb-app
spec:
containers:
- name: app
image: busybox:1.36
command: ["sleep", "3600"]
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: pdb-app-budget
spec:
minAvailable: 2
selector:
matchLabels:
app: pdb-app
EOF
kubectl rollout status deployment/pdb-protected-app
kubectl get pdb pdb-app-budget
# 2. 尝试排空 lima-node02(设置 5 秒超时),观察 PDB 当场报错拦截驱逐:Cannot evict pod as it would violate the pod's disruption budget!
kubectl drain lima-node02 --ignore-daemonsets --delete-emptydir-data --timeout=5s || true
# 3. 恢复 lima-node02 可调度状态(uncordon)
kubectl uncordon lima-node02
kubectl delete pdb pdb-app-budget8. 实验点 3.7:节点故障挤占与 Descheduler 重平衡实战
原理回顾: 当 lima-node02 故障或被 cordon + drain 维护时,原本分布在 node01 和 node02 上的所有业务 Pod 都会集中挤到 lima-node01 上。 但是,当 lima-node02 恢复 Ready 后,原生的 kube-scheduler 不会主动把已经运行在 lima-node01 上的 Pod 迁回 lima-node02!这时只需运行一次 Descheduler(配置 RemoveDuplicates 或 RemovePodsViolatingTopologySpreadConstraint 策略),即可自动驱逐堆叠在单节点的重复副本,实现集群重新均衡!
步骤 1:模拟 lima-node02 维护排空,导致 4 个副本全部挤在 lima-node01
bash
# 1. 将 pdb-protected-app 扩容为 4 副本并加上 topologySpreadConstraints 软约束
cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: pdb-protected-app
spec:
replicas: 4
selector:
matchLabels:
app: pdb-app
template:
metadata:
labels:
app: pdb-app
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels:
app: pdb-app
containers:
- name: app
image: busybox:1.36
command: ["sleep", "3600"]
EOF
kubectl rollout status deployment/pdb-protected-app
kubectl get pods -l app=pdb-app -o wide
# 2. 排空 lima-node02,使 4 个副本全部被赶到 lima-node01 上
kubectl drain lima-node02 --ignore-daemonsets --delete-emptydir-data
kubectl get pods -l app=pdb-app -o wide
# 3. 恢复 lima-node02 上线(uncordon),观察 4 个 Pod 依然全部留守在 lima-node01,lima-node02 空载!
kubectl uncordon lima-node02
kubectl get pods -l app=pdb-app -o wide步骤 2:以 Job 形式运行官方 Descheduler,一键触发集群自动重平衡!
bash
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
name: descheduler-sa
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: descheduler-cluster-role
rules:
- apiGroups: [""]
resources: ["nodes", "pods"]
verbs: ["get", "watch", "list"]
- apiGroups: [""]
resources: ["pods/eviction"]
verbs: ["create"]
- apiGroups: ["events.k8s.io", ""]
resources: ["events"]
verbs: ["create", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: descheduler-cluster-role-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: descheduler-cluster-role
subjects:
- kind: ServiceAccount
name: descheduler-sa
namespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:
name: descheduler-policy
namespace: kube-system
data:
policy.yaml: |
apiVersion: "descheduler/v1alpha2"
kind: "DeschedulerPolicy"
profiles:
- name: Profile1
pluginConfig:
- name: "DefaultEvictor"
- name: "RemoveDuplicates"
plugins:
balance:
enabled:
- "RemoveDuplicates"
---
apiVersion: batch/v1
kind: Job
metadata:
name: descheduler-run-once
namespace: kube-system
spec:
template:
spec:
serviceAccountName: descheduler-sa
restartPolicy: Never
containers:
- name: descheduler
image: registry.k8s.io/descheduler/descheduler:v0.30.1
command: ["/bin/descheduler"]
args: ["--policy-config-file=/policy-dir/policy.yaml", "--v=3"]
volumeMounts:
- mountPath: /policy-dir
name: policy-volume
volumes:
- name: policy-volume
configMap:
name: descheduler-policy
EOF
kubectl wait --for=condition=complete job/descheduler-run-once -n kube-system --timeout=90s
kubectl logs job/descheduler-run-once -n kube-system | tail -n 15
# 再次查看 4 个 Pod 的分布:已重新恢复为 lima-node01 (2个) + lima-node02 (2个) 的完美均衡状态!
kubectl get pods -l app=pdb-app -o wide9. 实验三环境清理
bash
kubectl delete job descheduler-run-once -n kube-system --ignore-not-found
kubectl delete cm descheduler-policy -n kube-system --ignore-not-found
kubectl delete clusterrolebinding descheduler-cluster-role-binding --ignore-not-found
kubectl delete clusterrole descheduler-cluster-role --ignore-not-found
kubectl delete sa descheduler-sa -n kube-system --ignore-not-found
kubectl config set-context --current --namespace=default
kubectl delete namespace lab3-schedule