Skip to content

实验三:高级调度策略、拓扑均衡、优先级抢占与容灾驱逐实战 ​

0. 实验目标与环境基线 ​

本实验利用你的 Mac mini M4 + Lima 三节点集群(lima-master、lima-node01、lima-node02),通过给节点打上不同的硬件标签与可用区(topology.kubernetes.io/zone)标签,完整演练 Kubernetes 调度器的预选与优选、节点亲和性、Pod 间亲和与反亲和、污点容忍、Even Pod Spreading (maxSkew) 跨拓扑均匀分布、PDB 破坏预算防并发驱逐中断、PriorityClass 高优先级抢占,以及使用 limactl stop 模拟节点掉线与 Descheduler 重平衡。

包含的 7 个核心实操点 ​

  1. 实验点 3.1:节点标签管理、nodeSelector 与 NodeAffinity(硬亲和 required vs 软亲和 preferred 权重打分)
  2. 实验点 3.2:PodAffinity(同节点聚集减少网络跳数)与 PodAntiAffinity(跨节点反亲和打散容灾)
  3. 实验点 3.3:节点污点(NoSchedule / PreferNoSchedule / NoExecute)与 Pod Tolerations(含 tolerationSeconds 限时驱逐)
  4. 实验点 3.4:Even Pod Spreading(topologySpreadConstraints + maxSkew: 1)实现跨节点/跨可用区严格均匀分布
  5. 实验点 3.5:PriorityClass 高优先级 Pod 抢占(Preemption)低优先级 Pod 算力实验
  6. 实验点 3.6:配置 PDB (PodDisruptionBudget) 拦截 kubectl drain 防止维护期服务全灭
  7. 实验点 3.7:节点失联故障演练(limactl stop node02)与 Descheduler 重调度器自动恢复平衡验证

1. 实验前准备:为 lima-node01 和 lima-node02 规划拓扑域与硬件标签 ​

在 lima-master 上执行:

bash
limactl shell master

kubectl create namespace lab3-schedule
kubectl config set-context --current --namespace=lab3-schedule

# 为 lima-node01 和 lima-node02 打上模拟的可用区 (zone) 标签和磁盘/算力标签
kubectl label node lima-node01 topology.kubernetes.io/zone=zone-a disktype=nvme accelerator=gpu --overwrite
kubectl label node lima-node02 topology.kubernetes.io/zone=zone-b disktype=sata --overwrite

# 查看节点标签
kubectl get nodes -L topology.kubernetes.io/zone,disktype,accelerator

2. 实验点 3.1:nodeSelector 与 NodeAffinity 软硬亲和验证 ​

原理回顾:

  • requiredDuringSchedulingIgnoredDuringExecution(硬亲和):必须满足,否则 Pod 卡在 Pending。
  • preferredDuringSchedulingIgnoredDuringExecution(软亲和):优选阶段给匹配节点加 weight 分,若不满足也能退而求其次调度到其他节点。
bash
# 1. 部署一个要求硬亲和 zone-a/zone-b,同时软亲和优先选择 disktype=sata (即 lima-node02) 的 Pod
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: affinity-demo
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: topology.kubernetes.io/zone
            operator: In
            values: ["zone-a", "zone-b"]
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        preference:
          matchExpressions:
          - key: disktype
            operator: In
            values: ["sata"]
  containers:
  - name: pause
    image: registry.k8s.io/pause:3.10
EOF

kubectl wait --for=condition=Ready pod/affinity-demo --timeout=60s
# 验证它因软亲和 100 分加分而精准落在了 lima-node02 上!
kubectl get pod affinity-demo -o wide

3. 实验点 3.2:PodAffinity 聚集与 PodAntiAffinity 反亲和打散 ​

原理回顾:

  • 聚集(PodAffinity):将 Web 后端与本地 Redis 缓存调度到同一个 kubernetes.io/hostname 拓扑域,降低跨节点网络开销。
  • 打散(PodAntiAffinity):强制同一个 Deployment 的 2 个副本不得落在同一台物理机上。
bash
# 1. 部署 2 副本 Redis,使用硬反亲和确保 node01 和 node02 各跑 1 个
cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis-cache
spec:
  replicas: 2
  selector:
    matchLabels:
      app: redis-cache
  template:
    metadata:
      labels:
        app: redis-cache
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app: redis-cache
            topologyKey: kubernetes.io/hostname
      containers:
      - name: redis
        image: busybox:1.36
        command: ["sleep", "3600"]
EOF

kubectl rollout status deployment/redis-cache
# 验证 2 个 Pod 严格分散在 lima-node01 和 lima-node02
kubectl get pods -l app=redis-cache -o wide

4. 实验点 3.3:节点污点(NoSchedule vs NoExecute)与限时容忍驱逐 ​

原理回顾:

  • NoSchedule:新 Pod 不进驻,但节点上已运行的 Pod 不受影响。
  • NoExecute:不仅新 Pod 不进驻,节点上已运行的未容忍 Pod 会被立即驱逐(若配置了 tolerationSeconds: 10,则容忍 10 秒后被驱逐)。
bash
# 1. 在 lima-node02 上创建一个带 NoExecute 10秒限时容忍的 Pod
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: taint-victim
spec:
  nodeName: lima-node02
  tolerations:
  - key: "maintenance"
    operator: "Equal"
    value: "true"
    effect: "NoExecute"
    tolerationSeconds: 10
  containers:
  - name: app
    image: busybox:1.36
    command: ["sleep", "3600"]
EOF

kubectl wait --for=condition=Ready pod/taint-victim --timeout=60s

# 2. 先给 lima-node02 打上 NoSchedule 污点,验证 taint-victim 依然稳稳运行
kubectl taint nodes lima-node02 maintenance=true:NoSchedule
kubectl get pod taint-victim -o wide

# 3. 升级污点为 NoExecute,观察 taint-victim 撑过 10 秒容忍期后被自动驱逐销毁!
kubectl taint nodes lima-node02 maintenance=true:NoExecute --overwrite
echo "Waiting 12 seconds for tolerationSeconds to expire..."
sleep 12
kubectl get pod taint-victim

# 4. 实验完毕立即清除 lima-node02 上的测试污点(末尾加减号 -)
kubectl taint nodes lima-node02 maintenance:NoSchedule-
kubectl taint nodes lima-node02 maintenance:NoExecute-

5. 实验点 3.4:Even Pod Spreading (topologySpreadConstraints) 均匀分布 ​

原理回顾:之前我们学到 PodAntiAffinity 硬反亲和只允许每个节点最多 1 个 Pod(当副本数为 4、节点数为 2 时,第 3 和第 4 个 Pod 会卡在 Pending)。而使用 topologySpreadConstraints 并设置 maxSkew: 1,可以让 4 个副本在 zone-a(lima-node01)和 zone-b(lima-node02)上实现完美的 2 : 2 均匀分布!

bash
cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: even-spread-app
spec:
  replicas: 4
  selector:
    matchLabels:
      app: even-spread
  template:
    metadata:
      labels:
        app: even-spread
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: even-spread
      containers:
      - name: app
        image: busybox:1.36
        command: ["sleep", "3600"]
EOF

kubectl rollout status deployment/even-spread-app
# 验证 lima-node01 (zone-a) 恰好 2 个,lima-node02 (zone-b) 恰好 2 个!
kubectl get pods -l app=even-spread -o wide

# 清理该 Deployment 释放算力供下一节抢占实验使用
kubectl delete deployment redis-cache even-spread-app

6. 实验点 3.5:PriorityClass 高优先级抢占(Preemption)实战 ​

原理回顾:你的 lima-node01 和 lima-node02 各有 2 vCPU(即 2000m)。当低优先级 Pod 把两个节点的 CPU requests 账面额度占满后,此时提交一个高优先级(PriorityClass)Pod,Scheduler 会自动触发抢占(Preemption),驱逐低优先级 Pod 为核心业务腾出空间!

bash
# 1. 创建高低两个 PriorityClass
cat <<'EOF' | kubectl apply -f -
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority-core
value: 100000
globalDefault: false
description: "Core payment service priority"
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: low-priority-batch
value: 100
globalDefault: false
description: "Offline batch task priority"
EOF

# 2. 部署 2 个低优先级 Pod,每个申请 1200m CPU(由于每台 Worker 只有 2000m CPU,每台 Worker 只能放 1 个,从而占满两台 Worker)
cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: low-prio-filler
spec:
  replicas: 2
  selector:
    matchLabels:
      app: filler
  template:
    metadata:
      labels:
        app: filler
    spec:
      priorityClassName: low-priority-batch
      containers:
      - name: pause
        image: busybox:1.36
        command: ["sleep", "3600"]
        resources:
          requests:
            cpu: "1200m"
            memory: "128Mi"
EOF

kubectl rollout status deployment/low-prio-filler
kubectl get pods -l app=filler -o wide

# 3. 现在提交一个申请 1200m CPU 的高优先级 Pod,观察它抢占并踢走其中一个低优先级 Pod!
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: high-prio-vip
spec:
  priorityClassName: high-priority-core
  containers:
  - name: vip
    image: busybox:1.36
    command: ["sleep", "3600"]
    resources:
      requests:
        cpu: "1200m"
        memory: "128Mi"
EOF

sleep 8
# 观察 high-prio-vip 成功 Running,而其中一个 low-prio-filler 被抢占后变成 Pending!
kubectl get pods -o wide
kubectl get events --sort-by=.lastTimestamp | tail -n 10

# 清理抢占实验资源
kubectl delete pod high-prio-vip
kubectl delete deployment low-prio-filler
kubectl delete priorityclass high-priority-core low-priority-batch

7. 实验点 3.6:PDB (PodDisruptionBudget) 拦截节点排空(kubectl drain)保护 ​

原理回顾:当执行 kubectl drain 维护节点时,如果业务设置了 PodDisruptionBudget (minAvailable: 2),而当前总共只有 2 个副本在线,API Server 的 Eviction 接口会阻止同时把两个副本都杀掉,必须保持至少 2 个可用。

bash
# 1. 创建 2 副本业务,并配置 PDB 要求 minAvailable: 2(即不允许任何自愿中断导致可用数低于 2)
cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pdb-protected-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: pdb-app
  template:
    metadata:
      labels:
        app: pdb-app
    spec:
      containers:
      - name: app
        image: busybox:1.36
        command: ["sleep", "3600"]
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: pdb-app-budget
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: pdb-app
EOF

kubectl rollout status deployment/pdb-protected-app
kubectl get pdb pdb-app-budget

# 2. 尝试排空 lima-node02(设置 5 秒超时),观察 PDB 当场报错拦截驱逐:Cannot evict pod as it would violate the pod's disruption budget!
kubectl drain lima-node02 --ignore-daemonsets --delete-emptydir-data --timeout=5s || true

# 3. 恢复 lima-node02 可调度状态(uncordon)
kubectl uncordon lima-node02
kubectl delete pdb pdb-app-budget

8. 实验点 3.7:节点故障挤占与 Descheduler 重平衡实战 ​

原理回顾: 当 lima-node02 故障或被 cordon + drain 维护时,原本分布在 node01 和 node02 上的所有业务 Pod 都会集中挤到 lima-node01 上。 但是,当 lima-node02 恢复 Ready 后,原生的 kube-scheduler 不会主动把已经运行在 lima-node01 上的 Pod 迁回 lima-node02!这时只需运行一次 Descheduler(配置 RemoveDuplicates 或 RemovePodsViolatingTopologySpreadConstraint 策略),即可自动驱逐堆叠在单节点的重复副本,实现集群重新均衡!

步骤 1:模拟 lima-node02 维护排空,导致 4 个副本全部挤在 lima-node01 ​

bash
# 1. 将 pdb-protected-app 扩容为 4 副本并加上 topologySpreadConstraints 软约束
cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pdb-protected-app
spec:
  replicas: 4
  selector:
    matchLabels:
      app: pdb-app
  template:
    metadata:
      labels:
        app: pdb-app
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: ScheduleAnyway
        labelSelector:
          matchLabels:
            app: pdb-app
      containers:
      - name: app
        image: busybox:1.36
        command: ["sleep", "3600"]
EOF

kubectl rollout status deployment/pdb-protected-app
kubectl get pods -l app=pdb-app -o wide

# 2. 排空 lima-node02,使 4 个副本全部被赶到 lima-node01 上
kubectl drain lima-node02 --ignore-daemonsets --delete-emptydir-data
kubectl get pods -l app=pdb-app -o wide

# 3. 恢复 lima-node02 上线(uncordon),观察 4 个 Pod 依然全部留守在 lima-node01,lima-node02 空载!
kubectl uncordon lima-node02
kubectl get pods -l app=pdb-app -o wide

步骤 2:以 Job 形式运行官方 Descheduler,一键触发集群自动重平衡! ​

bash
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: descheduler-sa
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: descheduler-cluster-role
rules:
- apiGroups: [""]
  resources: ["nodes", "pods"]
  verbs: ["get", "watch", "list"]
- apiGroups: [""]
  resources: ["pods/eviction"]
  verbs: ["create"]
- apiGroups: ["events.k8s.io", ""]
  resources: ["events"]
  verbs: ["create", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: descheduler-cluster-role-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: descheduler-cluster-role
subjects:
- kind: ServiceAccount
  name: descheduler-sa
  namespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: descheduler-policy
  namespace: kube-system
data:
  policy.yaml: |
    apiVersion: "descheduler/v1alpha2"
    kind: "DeschedulerPolicy"
    profiles:
      - name: Profile1
        pluginConfig:
        - name: "DefaultEvictor"
        - name: "RemoveDuplicates"
        plugins:
          balance:
            enabled:
              - "RemoveDuplicates"
---
apiVersion: batch/v1
kind: Job
metadata:
  name: descheduler-run-once
  namespace: kube-system
spec:
  template:
    spec:
      serviceAccountName: descheduler-sa
      restartPolicy: Never
      containers:
      - name: descheduler
        image: registry.k8s.io/descheduler/descheduler:v0.30.1
        command: ["/bin/descheduler"]
        args: ["--policy-config-file=/policy-dir/policy.yaml", "--v=3"]
        volumeMounts:
        - mountPath: /policy-dir
          name: policy-volume
      volumes:
      - name: policy-volume
        configMap:
          name: descheduler-policy
EOF

kubectl wait --for=condition=complete job/descheduler-run-once -n kube-system --timeout=90s
kubectl logs job/descheduler-run-once -n kube-system | tail -n 15

# 再次查看 4 个 Pod 的分布:已重新恢复为 lima-node01 (2个) + lima-node02 (2个) 的完美均衡状态!
kubectl get pods -l app=pdb-app -o wide

9. 实验三环境清理 ​

bash
kubectl delete job descheduler-run-once -n kube-system --ignore-not-found
kubectl delete cm descheduler-policy -n kube-system --ignore-not-found
kubectl delete clusterrolebinding descheduler-cluster-role-binding --ignore-not-found
kubectl delete clusterrole descheduler-cluster-role --ignore-not-found
kubectl delete sa descheduler-sa -n kube-system --ignore-not-found

kubectl config set-context --current --namespace=default
kubectl delete namespace lab3-schedule