Kubernetes 1.35 Worker 节点标准化部署流程
在构建高可用 Kubernetes 集群的过程中，Worker 节点的标准化配置是确保集群稳定性与可维护性的关键一环。本文详细记录了基于 Ubuntu/Debian 系统、使用 Containerd 作为容器运行时、Kubernetes v1.35 的通用 Worker 节点部署全过程，并提供一键自动化脚本，适用于大规模节点快速上线。
环境说明
目标节点列表（GPU计算型）：
192.168.7.71 k8s-worker-gpu01
192.168.7.72 k8s-worker-gpu02
192.168.7.73 k8s-worker-gpu03
192.168.7.74 k8s-worker-gpu04
192.168.7.75 k8s-worker-gpu05
硬件规格：64 核 CPU / 512 GB 内存 / 256 GB SSD（启用 writeback 与 discard）+ 8 TB HDD（挂载于 /var/lib，同样启用 writeback 与 discard），文件系统为 XFS。
注：GPU 节点（如
k8s-worker-gpu01）可复用相同流程，仅在打标签阶段区分角色。
部署步骤详解
1. 禁用 Swap
Kubernetes 官方明确要求关闭 swap，否则 kubelet 将无法启动。
swapoff -a
sed -i '/ swap / s/^/#/' /etc/fstab
2. 内核模块与网络配置
启用 overlay 和 br_netfilter 模块以支持容器网络，并调整 sysctl 参数：
# /etc/modules-load.d/k8s.conf
overlay
br_netfilter
# /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
执行 sysctl --system 使配置生效。
禁用nouveau
cat <<EOF | sudo tee /etc/modprobe.d/blacklist-nouveau.conf
blacklist nouveau
blacklist nvidiafb
options nouveau modeset=0
EOF
# 需更新所有镜像
update-initramfs -u -k all
# 重启
reboot
3. 安装 Containerd 运行时
从预下载的二进制包安装 Containerd、runc 及 CNI 插件：
- 解压
containerd-2.2.1-linux-amd64.tar.gz到/usr/local - 安装
runc.amd64到/usr/local/sbin/runc - 解压 CNI 插件到
/opt/cni/bin
生成默认配置并修改关键参数：
# /etc/containerd/config.toml
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true # 必须设为 true 以匹配 kubelet 的 systemd 驱动
注册 systemd 服务并启动：
systemctl daemon-reload
systemctl enable --now containerd
4.安装nvidia驱动
踩了一堆坑，还是建议认真阅读文档，严格按照文档来
（实际上nvidia驱动在安装完kubelet，kubeadm后也可以，甚至加入集群后都可以）
https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html
https://docs.nvidia.com/datacenter/tesla/driver-installation-guide/index.html
https://docs.nvidia.com/cuda/cuda-installation-guide-linux/
https://github.com/NVIDIA/k8s-device-plugin#prerequisites
https://github.com/NVIDIA/k8s-device-plugin#quick-start
官网流程(我的操作系统版本是ubuntu24.04，所以$distro是ubuntu2404)
apt install linux-headers-$(uname -r)
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
dpkg -i cuda-keyring_1.1-1_all.deb
apt update
apt list nvidia-driver-pinning-*
# 输出。。。
nvidia-driver-pinning-570.211.01/unknown 570.211.01-0ubuntu1 all
nvidia-driver-pinning-570/unknown 570-1ubuntu3 all
nvidia-driver-pinning-580.105.08/unknown 580.105.08-0ubuntu1 all
nvidia-driver-pinning-580.126.09/unknown 580.126.09-1ubuntu1 all
nvidia-driver-pinning-580/unknown 580-1ubuntu3 all
nvidia-driver-pinning-590.44.01/unknown 590.44.01-0ubuntu1 all
nvidia-driver-pinning-590.48.01/unknown 590.48.01-0ubuntu1 all
nvidia-driver-pinning-590/unknown 590-1ubuntu3 all
# 直接安装最新版
apt install nvidia-driver-pinning-590.48.01
# https://docs.nvidia.com/datacenter/tesla/driver-installation-guide/recent-updates.html#recent-updates-pinning
# 开源版
# apt install nvidia-open
# 闭源版（我先装闭源的试试）
# apt install cuda-drivers
# 装完了才看到，可以装纯计算驱动。。。
# 开源版
# apt -V install libnvidia-compute nvidia-dkms-open
# 闭源版（应该装这个）
# apt -V install libnvidia-compute nvidia-dkms
# 验证
nvidia-smi
5. 安装 NVIDIA Container Runtime（nvidia-container-toolkit）
按照文档
https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html
sudo apt-get update && sudo apt-get install -y --no-install-recommends \
curl \
gnupg2
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
&& curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
apt update
export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.18.1-1
sudo apt-get install -y \
nvidia-container-toolkit=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
nvidia-container-toolkit-base=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
libnvidia-container-tools=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
libnvidia-container1=${NVIDIA_CONTAINER_TOOLKIT_VERSION}
# Configuring containerd (for Kubernetes)
nvidia-ctk runtime configure --runtime=containerd
# 输出
INFO[0000] Using config version 3
INFO[0000] Using CRI runtime plugin name "io.containerd.cri.v1.runtime"
INFO[0000] Wrote updated config to /etc/containerd/conf.d/99-nvidia.toml
INFO[0000] It is recommended that containerd daemon be restarted.
# 强制nvidia作为默认运行时
nvidia-ctk runtime configure --runtime=containerd --set-as-default
# 如果没有强制默认运行时，就需要给k8s的配置加上
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
name: nvidia
handler: nvidia
6. 重新加载并重启 containerd
systemctl daemon-reload
systemctl restart containerd
7. 配置 crictl
安装 crictl 并指向 Containerd socket：
# /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
debug: false
验证输出中应包含 "systemdCgroup": true。
8. 预拉取镜像
为避免因网络问题导致 Pod 启动失败，提前拉取并重命名用到的镜像：
ctr -n k8s.io images pull registry.aliyuncs.com/google_containers/pause:3.10.1 --platform linux/amd64
ctr -n k8s.io images tag \
registry.aliyuncs.com/google_containers/pause:3.10.1 \
registry.k8s.io/pause:3.10.1
ctr -n k8s.io images pull "registry.aliyuncs.com/google_containers/kube-proxy:v1.35.0" --platform linux/$(uname -m)
ctr -n k8s.io images tag registry.aliyuncs.com/google_containers/kube-proxy:v1.35.0 registry.k8s.io/kube-proxy:v1.35.0
9. 安装 kubelet 与 kubeadm
使用清华大学 Kubernetes 镜像源安装指定版本组件：
# 添加 GPG 密钥
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.35/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
# 配置 APT 源
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://mirrors.tuna.tsinghua.edu.cn/kubernetes/core:/stable:/v1.35/deb/ /" > /etc/apt/sources.list.d/kubernetes.list
apt update
apt install -y kubelet=1.35.0-1.1 kubeadm=1.35.0-1.1
apt-mark hold kubelet kubeadm
10. 加入集群
如果没有安装过可以先在 Kubernetes 集群中部署官方 Device Plugin，使 kubelet 能发现 GPU 资源。
kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.18.0/nvidia-device-plugin.yml
# 需要调整节点选择--只在gpu节点上运行
spec:
nodeSelector:
node-role.kubernetes.io/worker-gpu: ""
在 Control Plane 节点执行：
kubeadm token create --print-join-command
将输出的命令在 Worker 节点执行，完成注册。
11. 节点角色标记（可选）
为便于调度区分，可为节点打上自定义角色标签：
# 通用节点
kubectl label node k8s-worker-general01 node-role.kubernetes.io/worker-general=""
# GPU 节点
kubectl label node k8s-worker-gpu01 node-role.kubernetes.io/worker-gpu=""
# GPU 节点污点,保证普通pod不会调度到gpu节点上
kubectl taint node k8s-worker-gpu01 dedicated=gpu:NoSchedule
kubectl taint node k8s-worker-gpu02 dedicated=gpu:NoSchedule
kubectl taint node k8s-worker-gpu03 dedicated=gpu:NoSchedule
kubectl taint node k8s-worker-gpu04 dedicated=gpu:NoSchedule
kubectl taint node k8s-worker-gpu05 dedicated=gpu:NoSchedule
# 如果是需要gpu的pod需要容忍
tolerations:
- key: "dedicated"
operator: "Equal"
value: "gpu"
effect: "NoSchedule"
此时 kubectl get nodes 将显示清晰的角色标识。
Device Plugin 以 DaemonSet 形式部署，每个有 GPU 的节点应运行一个 Pod。
kubectl get pods -n kube-system -l name=nvidia-device-plugin-ds
NAME READY STATUS RESTARTS AGE
nvidia-device-plugin-daemonset-5sctq 1/1 Running 0 2m35s
终于成功了
kubectl describe node k8s-worker-gpu01 | grep nvidia.com/gpu
nvidia.com/gpu: 2
nvidia.com/gpu: 2
部署官网的gpu-pod测试
kubectl logs gpu-pod
[Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done
(安装过程需要多次重启,一键部署脚本正在优化)
宝剑锋从磨砺出,梅花香自苦寒来.