Kubernetes高可用集群搭建与安全加固实战-北京尧图网络

1. 项目概述在云原生技术蓬勃发展的今天Kubernetes简称K8s已成为容器编排领域的事实标准。最近我在生产环境中完成了一个高可用K8s集群的搭建项目采用3台Ubuntu服务器作为控制平面节点实现了真正意义上的三控合一架构。这个方案最特别之处在于全程采用证书体系实现全链路加密并通过双向认证机制确保集群通信安全。下面我将详细分享这次实战中的技术细节和踩坑经验。2. 环境准备与架构设计2.1 硬件资源配置我们选用3台配置相同的物理服务器CPU: 8核Intel Xeon内存: 32GB存储: 500GB SSD系统盘 2TB HDD数据盘网络: 双万兆网卡bonding模式4注意控制平面节点建议至少4核CPU和16GB内存etcd对磁盘IOPS要求较高SSD是必须的2.2 操作系统配置所有节点统一使用Ubuntu 22.04 LTS进行了以下基础配置# 禁用swap sudo swapoff -a sudo sed -i / swap / s/^/#/ /etc/fstab # 设置时区同步 sudo timedatectl set-timezone Asia/Shanghai sudo apt install -y chrony sudo systemctl enable --now chronyd # 加载内核模块 cat EOF | sudo tee /etc/modules-load.d/k8s.conf overlay br_netfilter EOF sudo modprobe overlay sudo modprobe br_netfilter3. 证书体系设计与实现3.1 PKI基础设施搭建高可用集群的安全基石是健全的PKI体系。我们采用cfssl工具链生成所有证书# 安装cfssl工具 wget https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssl_1.6.4_linux_amd64 -O cfssl chmod x cfssl sudo mv cfssl /usr/local/bin/ # CA配置示例 cat ca-config.json EOF { signing: { default: { expiry: 8760h }, profiles: { kubernetes: { usages: [signing, key encipherment, server auth, client auth], expiry: 8760h } } } } EOF3.2 双向认证实现要点服务端证书每个服务(如kube-apiserver)需要包含所有可能的访问域名/IP客户端证书每个组件(kubelet、controller-manager等)需要独立证书证书轮换设置kubelet的--rotate-certificates参数实现自动轮换4. 高可用控制平面部署4.1 负载均衡配置使用HAProxyKeepalived实现API Server负载均衡frontend k8s-api bind *:6443 mode tcp option tcplog default_backend k8s-api backend k8s-api mode tcp option tcp-check balance roundrobin server k8s-master-1 192.168.1.101:6443 check server k8s-master-2 192.168.1.102:6443 check server k8s-master-3 192.168.1.103:6443 check4.2 etcd集群部署关键配置参数# etcd.service关键参数 ExecStart/usr/local/bin/etcd \\ --name${HOSTNAME} \\ --cert-file/etc/etcd/kubernetes.pem \\ --key-file/etc/etcd/kubernetes-key.pem \\ --peer-cert-file/etc/etcd/kubernetes.pem \\ --peer-key-file/etc/etcd/kubernetes-key.pem \\ --trusted-ca-file/etc/etcd/ca.pem \\ --peer-trusted-ca-file/etc/etcd/ca.pem \\ --peer-client-cert-authtrue \\ --client-cert-authtrue5. 集群初始化与验证5.1 kubeadm配置模板apiVersion: kubeadm.k8s.io/v1beta3 kind: ClusterConfiguration kubernetesVersion: 1.28.2 controlPlaneEndpoint: k8s-api.example.com:6443 apiServer: certSANs: - k8s-api.example.com - 192.168.1.100 # VIP - 192.168.1.101 - 192.168.1.102 - 192.168.1.103 - 127.0.0.1 networking: podSubnet: 10.244.0.0/16 serviceSubnet: 10.96.0.0/12 etcd: external: endpoints: - https://192.168.1.101:2379 - https://192.168.1.102:2379 - https://192.168.1.103:2379 caFile: /etc/kubernetes/pki/etcd/ca.crt certFile: /etc/kubernetes/pki/etcd/client.crt keyFile: /etc/kubernetes/pki/etcd/client.key5.2 集群健康检查# 检查节点状态 kubectl get nodes -o wide # 检查组件状态 kubectl get cs # 证书有效期检查 openssl x509 -noout -dates -in /etc/kubernetes/pki/apiserver.crt # 网络连通性测试 kubectl run -it --rm --imagealpine test -- sh ping kubernetes.default6. 关键问题排查记录6.1 证书过期问题现象突然无法创建新资源但已有Pod运行正常排查journalctl -u kube-apiserver -n 100 | grep -i cert发现报错x509: certificate has expired or is not yet valid解决方案备份现有证书使用kubeadm alpha certs renew all重启控制平面组件6.2 etcd集群脑裂现象kubectl命令执行缓慢部分节点NotReady 排查etcdctl --endpointshttps://192.168.1.101:2379 \ --cacert/etc/etcd/ca.pem \ --cert/etc/etcd/kubernetes.pem \ --key/etc/etcd/kubernetes-key.pem endpoint status发现节点间数据不一致解决方案确定leader节点从leader节点执行数据备份移除异常节点后重新加入7. 安全加固建议RBAC配置为每个服务账户分配最小权限kubectl create role developer \ --verbget,list,watch \ --resourcepods,deployments kubectl create rolebinding dev-binding \ --roledeveloper \ --userdev-user网络策略启用NetworkPolicy插件apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny spec: podSelector: {} policyTypes: - Ingress - Egress审计日志配置apiserver审计apiServer: extraArgs: audit-log-path: /var/log/kubernetes/audit.log audit-policy-file: /etc/kubernetes/audit-policy.yaml audit-log-maxage: 30 audit-log-maxbackup: 10 audit-log-maxsize: 1008. 性能优化实践kubelet配置调优--max-pods150 --kube-api-qps50 --kube-api-burst100 --image-pull-progress-deadline30metcd性能优化--auto-compaction-retention1h --max-request-bytes33554432 --quota-backend-bytes8589934592调度器配置apiVersion: kubescheduler.config.k8s.io/v1beta3 kind: KubeSchedulerConfiguration profiles: - schedulerName: default-scheduler pluginConfig: - name: NodeResourcesFit args: scoringStrategy: type: LeastAllocated resources: - name: cpu weight: 1 - name: memory weight: 19. 运维监控方案9.1 监控体系架构数据采集层Node-exporterKube-state-metricscAdvisor存储层Prometheus ThanosVictoriaMetrics展示层GrafanaAlertmanager9.2 关键监控指标类别关键指标告警阈值节点node_memory_MemAvailable_bytes 10%API Serverapiserver_request_duration_secondsp99 1setcdetcd_disk_wal_fsync_duration_secondsp99 500ms网络container_network_receive_bytes_total突增200%10. 灾备与恢复策略10.1 集群备份方案etcd定期快照ETCDCTL_API3 etcdctl --endpointshttps://127.0.0.1:2379 \ --cacert/etc/etcd/ca.pem \ --cert/etc/etcd/kubernetes.pem \ --key/etc/etcd/kubernetes-key.pem \ snapshot save /backup/etcd-snapshot-$(date %Y%m%d).db资源声明备份kubectl get all --all-namespaces -o yaml cluster-state-$(date %Y%m%d).yaml10.2 灾难恢复流程恢复etcd数据ETCDCTL_API3 etcdctl snapshot restore /backup/etcd-snapshot.db \ --data-dir /var/lib/etcd-restore重建控制平面kubeadm init --ignore-preflight-errorsall \ --config /etc/kubernetes/kubeadm-config.yaml恢复工作节点kubeadm join k8s-api.example.com:6443 \ --token xxxx \ --discovery-token-ca-cert-hash sha256:xxxx11. 证书管理进阶技巧11.1 证书自动轮换配置kube-controller-manager--experimental-cluster-signing-duration87600h --feature-gatesRotateKubeletServerCertificatetruekubelet配置--rotate-certificates --cert-dir/var/lib/kubelet/pki11.2 自定义CA轮换准备新CA证书分阶段更新配置kubeadm alpha certs renew all --use-api --config /etc/kubernetes/kubeadm-config.yaml滚动重启组件12. 网络方案选型对比方案性能复杂度适用场景Calico高中需要网络策略Flannel中低简单场景Cilium极高高云原生环境Weave中中混合云我们最终选择Calico方案配置示例apiVersion: crd.projectcalico.org/v1 kind: IPPool metadata: name: default-ipv4-ippool spec: cidr: 10.244.0.0/16 ipipMode: Never natOutgoing: true nodeSelector: all()13. 存储方案集成13.1 CSI驱动选择本地存储OpenEBS LocalPV网络存储Ceph RBD云存储AWS EBS/Azure Disk13.2 存储类配置apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: fast-disks provisioner: pd.csi.storage.gke.io parameters: type: pd-ssd replication-type: none volumeBindingMode: WaitForFirstConsumer allowVolumeExpansion: true14. 版本升级策略先升级kubeadmapt-get update apt-get install -y kubeadm1.28.3-00控制平面升级kubeadm upgrade plan kubeadm upgrade apply v1.28.3工作节点升级kubectl drain node --ignore-daemonsets apt-get update apt-get install -y kubelet1.28.3-00 kubectl1.28.3-00 systemctl restart kubelet kubectl uncordon node15. 服务暴露方案15.1 Ingress控制器选型Nginx Ingress功能全面社区支持好Traefik动态配置云原生友好Contour基于Envoy支持gRPC15.2 LoadBalancer方案apiVersion: v1 kind: Service metadata: name: my-service spec: selector: app: my-app ports: - protocol: TCP port: 80 targetPort: 9376 type: LoadBalancer16. 安全扫描与合规镜像扫描trivy image --security-checks vuln my-image:latest配置检查kube-bench run --targets master,node,etcd网络策略验证kubectl-network-policy-visualizer17. 成本优化实践节点自动伸缩apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: my-app-hpa spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: my-app minReplicas: 2 maxReplicas: 10 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 50资源请求优化kubectl top pods --containers kubectl get pods --outputjsonpath{range .items[*]}{.metadata.name}{\t}{.spec.containers[*].resources}{\n}{end}18. 多集群管理方案Kubefed架构kubefedctl join cluster1 --cluster-contextcluster1 \ --host-cluster-contextcluster0 \ --v2集群状态同步apiVersion: types.kubefed.io/v1beta1 kind: FederatedDeployment metadata: name: my-app namespace: my-ns spec: placement: clusters: - name: cluster1 - name: cluster2 template: spec: containers: - image: my-app:latest name: my-app19. 日志收集方案19.1 EFK Stack部署Fluentd配置source type tail path /var/log/containers/*.log pos_file /var/log/fluentd-containers.log.pos tag kubernetes.* read_from_head true parse type json time_format %Y-%m-%dT%H:%M:%S.%NZ /parse /source19.2 Loki方案scrape_configs: - job_name: kubernetes-pods kubernetes_sd_configs: - role: pod relabel_configs: - source_labels: [__meta_kubernetes_pod_annotation_loki_logs] action: keep regex: true20. 持续交付流水线GitOps工作流flux bootstrap github \ --ownermy-org \ --repositorymy-repo \ --branchmain \ --path./clusters/production \ --personalTekton Pipeline示例apiVersion: tekton.dev/v1beta1 kind: Pipeline metadata: name: build-deploy spec: workspaces: - name: shared-data tasks: - name: build taskRef: name: buildah workspaces: - name: source workspace: shared-data

Kubernetes高可用集群搭建与安全加固实战

相关推荐

TVBoxOSC：每天自动更新的两套TVBox构建，拿来即用

C++ STL核心组件：容器、迭代器与算法设计解析

MATLAB脑电信号预处理与噪声消除技术详解

更多资讯

Comprehensive Rust 生命周期实战：用 Rust 手写一个零拷贝 Protobuf 二进制解析器（附完整解答与测试）

Milvus QueryNode v2 重构设计解读：Delegator/Worker 角色分离与无 delta channel 的删除转发架构

使用 claude-task-master 的 Hamster Brief 端到端任务执行工作流

LocalAI 模型别名（Model Aliases）完全指南：零客户端改动的模型重定向与灰度切换

两级式SST技术：高效电力转换与工业应用解析

基于SpringBoot的宠物健康管理系统的设计与实现毕业设计项目源码

今日更新

CopilotKit Vue Demo 与 React Demo 的 Parity 实现指南：路由、运行时端点与交互语义的全框架对齐

Harbor RBAC 实战：db_auth 模式下非管理员用户管理项目成员的完整验证与权限模型解析

STM32平衡车纯平衡程序从拆包到参数整定的完整调试指南

本周精选

超人会飞不算本事：系统稳定依赖清晰规则与边界设计

超人VS蜘蛛侠：拆解超级IP的影响力与传播方法论

基于CNN的调制信号识别：MATLAB实现时频图分类实战

本月热门

基于YOLO的焊缝缺陷检测：从数据标注到工业部署全流程实践

分布式服务的决策记录

创业产品的首版范围

读完文章，想聊聊您的网站？