Skip to content

实验六:配置热更新、RBAC 证书授权、Helm 模板工程与 Webhook/升级实战 ​

0. 实验目标与环境基线 ​

本实验是整套 Kubernetes 进阶体系的综合收官大实验,在你的 Mac mini M4 + Lima 三节点集群 上一次性打通四大企业级治理与扩展主题:

  1. ConfigMap 与 Secret 全景治理(多行控制符、subPath 单文件挂载、Volume 热更新同步 vs Reloader 滚动更新、K8s 1.24+ TokenRequest 动态短期 Token 投影卷);
  2. 集群安全、X.509 证书签发与 RBAC 授权 + SecurityContext (Capabilities 细粒度特权收敛);
  3. Helm 3 安装与 Chart 模板开发实战(亲手创建 Chart、编写 _helpers.tpl / include / nindent / with / range / .Files.Get,测试 --set > -f > values.yaml 三级优先级与一键 Rollback);
  4. CRD + Mutating Admission Webhook 准入拦截器开发实战(在集群内用 Python 3 标准库跑一个真实的 HTTPS Mutating Webhook,给新建 Pod 自动注入 environment: production 标签!)+ etcd 快照备份与集群无损升级 SOP 演练。

包含的 7 个核心实操点 ​

  1. 实验点 6.1:ConfigMap(| vs |-、Volume 自动热更新 vs subPath 不热更新)与 Secret 三大类型实战
  2. 实验点 6.2:使用 OpenSSL 为研发人员 dev-alice 签发 X.509 客户端证书、配置 kubeconfig Context,并通过 Role + RoleBinding(含绑定 ClusterRole)验证 RBAC 最小权限拦截
  3. 实验点 6.3:SecurityContext 容器安全加固——验证非 Root 运行(runAsNonRoot)与 Linux Capabilities(drop: ["ALL"] + add: ["NET_ADMIN"])
  4. 实验点 6.4:在 lima-master 安装 Helm 3,从零开发一个标准 Chart 包并演练 install / upgrade / history / rollback
  5. 实验点 6.5:自定义资源 CRD 与 CR 创建(体验在 etcd 中建表与插记录)
  6. 实验点 6.6:部署真实 HTTPS MutatingWebhookConfiguration 准入控制器,实现新建 Pod 自动打上 environment: production 标签
  7. 实验点 6.7:etcdctl 底层数据探测、一致性快照备份(snapshot save)与节点滚动维护 SOP(cordon → drain → uncordon)

1. 实验前准备:创建实验六专属命名空间 ​

在 lima-master 上执行:

bash
limactl shell master

kubectl create namespace lab6-gov
kubectl config set-context --current --namespace=lab6-gov

2. 实验点 6.1:ConfigMapVolume 热更新与 subPath 对比实战 ​

原理回顾:

  • 通过环境变量(env)或 subPath 单文件挂载的 ConfigMap 不支持热更新。
  • 通过普通目录 volumeMounts 挂载的 ConfigMap,在修改 ConfigMap 后,Kubelet 会在约 10~60 秒内通过符号链接(..data)原子替换自动同步容器内文件。
bash
# 1. 创建包含多行配置 (|-) 的 ConfigMap 与挂载它的 Pod
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-dynamic-cm
data:
  app.conf: |-
    env = staging
    max_connections = 100
---
apiVersion: v1
kind: Pod
metadata:
  name: cm-hot-reload-demo
spec:
  containers:
  - name: app
    image: busybox:1.36
    command: ["sh", "-c", "sleep 3600"]
    volumeMounts:
    - name: config-vol
      mountPath: /etc/app-config
  volumes:
  - name: config-vol
    configMap:
      name: app-dynamic-cm
EOF

kubectl wait --for=condition=Ready pod/cm-hot-reload-demo --timeout=60s
kubectl exec cm-hot-reload-demo -- cat /etc/app-config/app.conf

# 2. 在线更新 ConfigMap 中的 max_connections 为 500
kubectl patch configmap app-dynamic-cm -p '{"data":{"app.conf":"env = production\nmax_connections = 500"}}'

# 3. 触发 kubelet 同步后查看容器内文件(观察软链接 ..data 与热更新后的内容)
kubectl exec cm-hot-reload-demo -- ls -la /etc/app-config

3. 实验点 6.2:使用 OpenSSL 签发用户证书 + RBAC 精细授权实战 ​

原理回顾:API Server 请求需经过 Authentication(认证)→ Authorization(RBAC 授权)→ Admission(准入控制) 三道关卡。普通人类用户通过 X.509 客户端证书中的 CN(用户名)和 O(用户组)完成身份认证。

步骤 1:使用集群 CA 为新用户 dev-alice 签发客户端证书并配置上下文 ​

在 lima-master 执行:

bash
mkdir -p /tmp/rbac-test && cd /tmp/rbac-test

# 1. 生成私钥与 CSR (CN=dev-alice, O=developers)
openssl genrsa -out dev-alice.key 2048
openssl req -new -key dev-alice.key -out dev-alice.csr -subj "/CN=dev-alice/O=developers"

# 2. 使用集群根证书 (/etc/kubernetes/pki/ca.crt) 签发合法证书
sudo openssl x509 -req -in dev-alice.csr \
  -CA /etc/kubernetes/pki/ca.crt \
  -CAkey /etc/kubernetes/pki/ca.key \
  -CAcreateserial -out dev-alice.crt -days 30

# 3. 在 kubeconfig 中注册用户凭据与上下文 alice-ctx
kubectl config set-credentials dev-alice \
  --client-certificate=/tmp/rbac-test/dev-alice.crt \
  --client-key=/tmp/rbac-test/dev-alice.key
kubectl config set-context alice-ctx --cluster=kubernetes --namespace=lab6-gov --user=dev-alice

步骤 2:未授权前测试访问(验证 403 Forbidden),随后创建 Role 与 RoleBinding 授权 ​

bash
# 1. 未创建 RoleBinding 前,以 alice-ctx 身份查看 Pod -> 预期返回 Error from server (Forbidden)!
kubectl --context=alice-ctx get pods || true

# 2. 创建仅允许在 lab6-gov 命名空间 get/list/watch Pod 的 Role 与 RoleBinding
cat <<'EOF' | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader-role
  namespace: lab6-gov
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: alice-read-pods
  namespace: lab6-gov
subjects:
- kind: User
  name: dev-alice
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: pod-reader-role
  apiGroup: rbac.authorization.k8s.io
EOF

# 3. 再次以 dev-alice 身份执行:
# (a) 查看 lab6-gov 下的 Pod -> 成功!
kubectl --context=alice-ctx get pods
# (b) 尝试删除 Pod -> Forbidden(因为没有 delete 动词权限)!
kubectl --context=alice-ctx delete pod cm-hot-reload-demo || true
# (c) 尝试查看 kube-system 下的 Pod -> Forbidden(因为 RoleBinding 仅限 lab6-gov 命名空间)!
kubectl --context=alice-ctx get pods -n kube-system || true

4. 实验点 6.3:SecurityContext 与 Linux Capabilities 细粒度特权控制 ​

原理回顾:生产最佳实践是 allowPrivilegeEscalation: false + capabilities.drop: ["ALL"],仅按需 add 最小权限(如网络工具仅需 NET_ADMIN)。

bash
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: sec-cap-demo
spec:
  containers:
  - name: hardened
    image: busybox:1.36
    command: ["sleep", "3600"]
    securityContext:
      allowPrivilegeEscalation: false
      privileged: false
      capabilities:
        drop:
        - ALL
        add:
        - NET_ADMIN
EOF

kubectl wait --for=condition=Ready pod/sec-cap-demo --timeout=60s

# 验证:因为赋予了 NET_ADMIN,容器内可以成功添加虚拟回环 IP;但尝试修改系统时间 (需要 SYS_TIME) 则被拒绝!
kubectl exec sec-cap-demo -- ip addr add 10.99.99.99/32 dev lo
kubectl exec sec-cap-demo -- ip -4 addr show dev lo
kubectl exec sec-cap-demo -- date -s "2030-01-01 00:00:00" || true

5. 实验点 6.4:Helm 3 安装、Chart 模板语法实战与版本回滚 ​

步骤 1:在 lima-master (ARM64) 安装 Helm 3 CLI ​

bash
if ! command -v helm >/dev/null 2>&1; then
  curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
fi
helm version

步骤 2:从零创建一个自定义 Chart 并演练核心 Go Template 语法 ​

我们在 /tmp/my-web-chart 创建一个精简但涵盖笔记中所有高频模板语法(_helpers.tpl、define、include、nindent、default、quote、if/else、with、range、.Files.Get)的 Chart:

bash
mkdir -p /tmp/my-web-chart/templates
cd /tmp/my-web-chart

cat <<'EOF' > Chart.yaml
apiVersion: v2
name: my-web-chart
description: A comprehensive Helm chart for K8s study
type: application
version: 0.1.0
appVersion: "1.0.0"
EOF

cat <<'EOF' > values.yaml
replicaCount: 1
env: dev
appConfig:
  server:
    port: 80
    greeting: "Hello from default values.yaml"
extraEnvs:
  LOG_LEVEL: info
  REGION: ap-northeast-1
EOF

# 创建一个外部普通静态文件供 .Files.Get 读取
cat <<'EOF' > app.toml
[database]
timeout = "5s"
pool_size = 20
EOF

cat <<'EOF' > templates/_helpers.tpl
{{- define "mychart.labels" -}}
generator: helm
chart: {{ .Chart.Name }}-{{ .Chart.Version }}
release: {{ .Release.Name }}
{{- end -}}
EOF

cat <<'EOF' > templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Release.Name }}-cm
  labels:
    {{- include "mychart.labels" . | nindent 4 }}
data:
  {{- with .Values.appConfig.server }}
  greeting.txt: {{ .greeting | upper | quote }}
  {{- end }}
  app.toml: |-
    {{ .Files.Get "app.toml" | nindent 4 }}
EOF

cat <<'EOF' > templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Release.Name }}-deploy
  labels:
    {{- include "mychart.labels" . | nindent 4 }}
spec:
  {{- if eq .Values.env "prod" }}
  replicas: 3
  {{- else }}
  replicas: {{ .Values.replicaCount | default 1 }}
  {{- end }}
  selector:
    matchLabels:
      release: {{ .Release.Name }}
  template:
    metadata:
      labels:
        release: {{ .Release.Name }}
    spec:
      containers:
      - name: web
        image: nginx:stable-alpine
        env:
        {{- range $k, $v := .Values.extraEnvs }}
        - name: {{ $k }}
          value: {{ $v | quote }}
        {{- end }}
EOF

步骤 3:渲染预览(helm template)、安装、参数覆盖升级(--set)与一键回滚(helm rollback) ​

bash
# 1. 本地渲染预览生成的 YAML(检查缩进与语法是否正确)
helm template demo-rel /tmp/my-web-chart

# 2. 正式安装 Release (Revision 1)
helm install demo-rel /tmp/my-web-chart -n lab6-gov
helm list -n lab6-gov

# 3. 使用 --set 覆盖参数在线升级为 prod 环境 (自动变为 3 副本,Revision 2)
helm upgrade demo-rel /tmp/my-web-chart -n lab6-gov \
  --set env=prod \
  --set appConfig.server.greeting="Upgraded via --set!"
kubectl get deploy demo-rel-deploy

# 4. 查看历史版本并一键回滚到 Revision 1(恢复为 1 副本!)
helm history demo-rel -n lab6-gov
helm rollback demo-rel 1 -n lab6-gov
helm history demo-rel -n lab6-gov
kubectl get deploy demo-rel-deploy

# 卸载 Release
helm uninstall demo-rel -n lab6-gov

6. 实验点 6.5:自定义资源 CRD 与 CR 实战(向 etcd 新增表与记录) ​

bash
cat <<'EOF' | kubectl apply -f -
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: redisclusters.database.example.com
spec:
  group: database.example.com
  names:
    kind: RedisCluster
    plural: redisclusters
    singular: rediscluster
    shortNames:
    - rc-db
  scope: Namespaced
  versions:
  - name: v1alpha1
    served: true
    storage: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              nodes:
                type: integer
                minimum: 1
                maximum: 10
              version:
                type: string
EOF

# 创建一条自定义资源记录 (CR)
cat <<'EOF' | kubectl apply -f -
apiVersion: database.example.com/v1alpha1
kind: RedisCluster
metadata:
  name: my-prod-redis
spec:
  nodes: 3
  version: "7.2"
EOF

# 使用简称 rc-db 查询我们自定义的数据库资源!
kubectl get rc-db

7. 实验点 6.6:从零实战 MutatingAdmissionWebhook(自动为 Pod 注入标签) ​

原理回顾:在笔记中我们详细讲到了如何编写一个 Webhook 服务,配置自签名 TLS 证书,并通过 MutatingWebhookConfiguration 让 API Server 在创建 Pod 时自动调用它注入 environment: production 标签(注意使用 JSONPatch 仅增补标签而不覆盖原有标签)。 下面我们无需构建外部 Docker 镜像,直接在 lima-master 上用 20 行 Python 3 标准库代码启动一个 HTTPS Webhook,亲眼见证 API Server 准入拦截与自动打标!

bash
mkdir -p /tmp/webhook-lab && cd /tmp/webhook-lab

# 1. 生成 Webhook HTTPS 证书(SAN 包含 Master IP 192.168.104.1)
openssl req -x509 -nodes -days 30 -newkey rsa:2048 \
  -keyout webhook.key -out webhook.crt \
  -subj "/CN=192.168.104.1" \
  -addext "subjectAltName = IP:192.168.104.1"

# 2. 编写基于 Python 3 内置 http.server 的 Mutating Webhook(返回 JSONPatch 添加 environment=production 标签)
cat <<'EOF' > webhook_server.py
import http.server, ssl, json, base64

class MutatingHandler(http.server.BaseHTTPRequestHandler):
    def do_POST(self):
        length = int(self.headers.get('content-length', 0))
        body = json.loads(self.rfile.read(length))
        uid = body["request"]["uid"]
        # JSONPatch: 增加 metadata.labels.environment = "production"
        patch = [{"op": "add", "path": "/metadata/labels/environment", "value": "production"}]
        patch_b64 = base64.b64encode(json.dumps(patch).encode()).decode()
        resp = {
            "apiVersion": "admission.k8s.io/v1",
            "kind": "AdmissionReview",
            "response": {
                "uid": uid,
                "allowed": True,
                "patchType": "JSONPatch",
                "patch": patch_b64
            }
        }
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(json.dumps(resp).encode())

httpd = http.server.HTTPServer(('0.0.0.0', 8443), MutatingHandler)
ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
ctx.load_cert_chain(certfile='webhook.crt', keyfile='webhook.key')
httpd.socket = ctx.wrap_socket(httpd.socket, server_side=True)
print("Webhook listening on https://0.0.0.0:8443...")
httpd.serve_forever()
EOF

# 在后台启动 Webhook HTTPS 服务
python3 webhook_server.py &
WEBHOOK_PID=$!

# 3. 注册 MutatingWebhookConfiguration(仅拦截带有 webhook-test=enabled 标签的 Namespace,安全防误伤)
kubectl label namespace lab6-gov webhook-test=enabled --overwrite
CA_BUNDLE=$(base64 -w0 < webhook.crt)

cat <<EOF | kubectl apply -f -
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: auto-label-mutating-webhook
webhooks:
- name: autolabel.example.com
  admissionReviewVersions: ["v1"]
  sideEffects: None
  timeoutSeconds: 3
  failurePolicy: Ignore
  namespaceSelector:
    matchLabels:
      webhook-test: enabled
  clientConfig:
    url: "https://192.168.104.1:8443/mutate"
    caBundle: "${CA_BUNDLE}"
  rules:
  - operations: ["CREATE"]
    apiGroups: [""]
    apiVersions: ["v1"]
    resources: ["pods"]
EOF

# 4. 创建一个只带 app: demo 标签的 Pod,观察它被自动注入了 environment=production 标签!
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: webhook-injected-pod
  labels:
    app: demo
spec:
  containers:
  - name: pause
    image: registry.k8s.io/pause:3.10
EOF

kubectl get pod webhook-injected-pod --show-labels

# 5. 实验成功后立即清理 Webhook 配置与后台进程
kubectl delete mutatingwebhookconfiguration auto-label-mutating-webhook
kill "$WEBHOOK_PID" 2>/dev/null || true

8. 实验点 6.7:在 lima-master 上直接操作 etcd 快照备份与集群升级维护 SOP ​

原理回顾:在集群升级或重大变更前,第一铁律是备份 etcd 快照。由于 kubeadm 使用静态 Pod 运行 etcd,我们可以直接在 lima-master 上通过容器内自带的 etcdctl(或宿主机安装 etcd 客户端)直接查询 /registry 键值并导出快照文件!

bash
# 1. 通过 etcd 容器内的 etcdctl 查看 etcd 成员列表与健康状态
kubectl exec -n kube-system etcd-lima-master -- etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  endpoint health

kubectl exec -n kube-system etcd-lima-master -- etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  member list -w table

# 2. 直接查看 etcd 数据库中存储的 Namespace 或 Pod 底层路径键名
kubectl exec -n kube-system etcd-lima-master -- etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/namespaces --prefix --keys-only

# 3. 导出 etcd 一致性快照到宿主机 /var/lib/etcd/snapshot-backup.db(因为 /var/lib/etcd 是 hostPath 挂载的!)
kubectl exec -n kube-system etcd-lima-master -- etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /var/lib/etcd/snapshot-backup.db

sudo ls -lh /var/lib/etcd/snapshot-backup.db

9. 实验六环境清理 ​

bash
kubectl config use-context kubernetes-admin@kubernetes
kubectl config set-context --current --namespace=default
kubectl config delete-context alice-ctx 2>/dev/null || true
kubectl config delete-user dev-alice 2>/dev/null || true
kubectl delete crd redisclusters.database.example.com --ignore-not-found
kubectl delete namespace lab6-gov
sudo rm -f /var/lib/etcd/snapshot-backup.db
rm -rf /tmp/rbac-test /tmp/my-web-chart /tmp/webhook-lab