1. 为什么需要K8s高可用与Ingress结合方案在云原生架构中Kubernetes集群的高可用性只是基础设施层面的保障。真正要实现业务层面的高可用必须将集群控制平面、工作节点与流量入口统一纳入容灾体系。这就是为什么在完成K8s集群部署后我们需要立即引入Ingress控制器作为流量调度枢纽。去年我在某电商平台的迁移项目中就遇到过惨痛教训虽然集群本身配置了多Master节点但由于Nginx Ingress采用单副本部署当该节点突发宕机时整个平台的对外API服务中断了近40分钟。这个案例让我深刻认识到完整的K8s高可用方案必须包含以下三个层级控制平面高可用通过多个Master节点组成etcd集群避免单点故障工作节点高可用合理设置Pod反亲和性确保关键业务分散在不同物理节点流量入口高可用Ingress控制器需要实现多副本部署自动故障转移本次部署选择Nginx Ingress Controller作为实现方案主要基于以下考量社区活跃度高与Kubernetes版本兼容性好支持蓝绿部署、灰度发布等高级流量管理功能性能优异单个实例可处理每秒5000请求配置灵活可通过Annotations实现精细化控制2. 前置环境检查与组件规划2.1 集群健康状态验证在开始部署Ingress前我们需要确认基础集群的运行状态符合预期。执行以下命令进行验证# 检查节点Ready状态 kubectl get nodes -o wide # 检查核心组件健康状况 kubectl get cs # 验证DNS服务是否正常 kubectl run -it --rm --restartNever dig-test --imageinfoblox/dnstools:latest -- dig kubernetes.default.svc.cluster.local预期输出应显示所有节点状态为Ready核心组件Healthy且能够正常解析集群内域名。如果发现异常需要优先排查网络插件如Calico或DNS服务CoreDNS的配置问题。2.2 网络拓扑规划建议合理的网络规划能有效避免后续的性能瓶颈。根据生产环境经验我推荐以下部署模式组件副本数部署方式节点选择建议Ingress Controller3DaemonSet专属Ingress节点组CoreDNS3Deployment分散在不同可用区业务应用2DeploymentPodAntiAffinity与Ingress节点隔离这种架构设计可以带来三个优势DaemonSet确保每个物理节点运行一个Ingress实例避免端口冲突专用节点组避免业务Pod与流量转发组件资源竞争反亲和性策略降低单点故障风险3. Ingress控制器部署实战3.1 使用Helm部署Nginx Ingress官方推荐的安装方式是使用Helm Chart这比直接apply manifest文件更便于后续升级和维护。以下是具体操作步骤# 添加Ingress-Nginx仓库 helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx helm repo update # 定制化values配置文件 cat ingress-values.yaml EOF controller: replicaCount: 3 hostNetwork: true kind: DaemonSet nodeSelector: node-role.kubernetes.io/ingress: true tolerations: - key: node-role.kubernetes.io/ingress operator: Exists effect: NoSchedule service: type: ClusterIP metrics: enabled: true serviceMonitor: enabled: true EOF # 安装Ingress-Nginx helm install ingress-nginx ingress-nginx/ingress-nginx \ -n ingress-nginx --create-namespace \ -f ingress-values.yaml关键参数解析hostNetwork: true直接使用主机网络提升网络性能kind: DaemonSet确保每个节点运行一个PodnodeSelector限制只在标记为ingress的节点上运行metrics.enabled开启Prometheus指标暴露3.2 节点标记与网络配置为专用节点打上标签并配置内核参数优化网络性能# 标记Ingress专用节点 kubectl label nodes node1 node2 node3 node-role.kubernetes.io/ingresstrue # 调整内核参数 echo net.core.somaxconn 32768 net.ipv4.tcp_max_syn_backlog 8192 net.ipv4.tcp_tw_reuse 1 fs.file-max 2097152 | sudo tee /etc/sysctl.d/99-ingress.conf # 应用参数修改 sudo sysctl -p /etc/sysctl.d/99-ingress.conf重要提示如果节点已经运行其他Pod需要先排空节点kubectl drain node-name --ignore-daemonsets --delete-emptydir-data4. 高可用业务部署实践4.1 示例应用部署与Ingress配置下面通过一个实际案例演示如何部署高可用业务。我们以Nginx作为示例应用# nginx-deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-demo spec: replicas: 3 selector: matchLabels: app: nginx-demo template: metadata: labels: app: nginx-demo spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - nginx-demo topologyKey: kubernetes.io/hostname containers: - name: nginx image: nginx:1.25 ports: - containerPort: 80 --- # nginx-ingress.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: nginx-demo annotations: nginx.ingress.kubernetes.io/affinity: cookie nginx.ingress.kubernetes.io/affinity-mode: persistent spec: ingressClassName: nginx rules: - host: demo.example.com http: paths: - path: / pathType: Prefix backend: service: name: nginx-demo port: number: 80关键配置说明podAntiAffinity确保Pod分散在不同节点affinity注解实现会话保持ingressClassName指定使用的Ingress控制器4.2 高级流量管理技巧在生产环境中我们通常需要更精细的流量控制蓝绿发布annotations: nginx.ingress.kubernetes.io/canary: true nginx.ingress.kubernetes.io/canary-weight: 20基于Header的路由annotations: nginx.ingress.kubernetes.io/canary-by-header: X-Env nginx.ingress.kubernetes.io/canary-by-header-value: staging限流保护annotations: nginx.ingress.kubernetes.io/limit-rpm: 100 nginx.ingress.kubernetes.io/limit-burst: 505. 监控与自动化运维5.1 Prometheus监控集成Ingress控制器的监控指标对于容量规划至关重要。以下是集成Prometheus的配置示例# ingress-monitor.yaml apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: ingress-nginx labels: release: prometheus spec: selector: matchLabels: app.kubernetes.io/instance: ingress-nginx endpoints: - port: metrics interval: 15s path: /metrics关键指标告警规则建议nginx_ingress_controller_requests请求量突增/突降nginx_ingress_controller_nginx_process_requests_total5xx错误率nginx_ingress_controller_connections连接数接近上限5.2 自动化扩缩容配置结合HPA实现Ingress控制器自动扩缩容apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: ingress-nginx spec: scaleTargetRef: apiVersion: apps/v1 kind: DaemonSet name: ingress-nginx-controller minReplicas: 3 maxReplicas: 10 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 70 - type: External external: metric: name: nginx_ingress_controller_requests_per_second selector: matchLabels: app: ingress-nginx target: type: AverageValue averageValue: 10006. 故障排查与性能优化6.1 常见问题速查表故障现象可能原因解决方案502 Bad Gateway后端服务不可用检查Service selector与Pod标签是否匹配413 Request Entity Too Large客户端上传大文件添加注解nginx.ingress.kubernetes.io/proxy-body-size: 20m连接超时节点网络插件问题检查Calico/IPVS配置验证节点间网络连通性证书过期Lets Encrypt证书未自动续期检查cert-manager日志验证ClusterIssuer配置6.2 性能调优实战经验经过多次压力测试我总结了以下性能优化组合内核参数调优# 增加连接跟踪表大小 echo 262144 /proc/sys/net/nf_conntrack_max # 提高本地端口范围 echo 1024 65535 /proc/sys/net/ipv4/ip_local_port_rangeIngress配置优化annotations: nginx.ingress.kubernetes.io/upstream-keepalive-connections: 100 nginx.ingress.kubernetes.io/upstream-keepalive-timeout: 60 nginx.ingress.kubernetes.io/proxy-buffer-size: 16k资源限制建议resources: limits: cpu: 2 memory: 2Gi requests: cpu: 500m memory: 512Mi在电商大促期间这套配置成功支撑了单Ingress实例每秒8000请求的流量峰值。关键是要根据实际监控数据持续调整参数特别是upstream-keepalive-connections和proxy-buffer-size这两个对性能影响最大的参数。
1. 为什么需要K8s高可用与Ingress结合方案在云原生架构中Kubernetes集群的高可用性只是基础设施层面的保障。真正要实现业务层面的高可用必须将集群控制平面、工作节点与流量入口统一纳入容灾体系。这就是为什么在完成K8s集群部署后我们需要立即引入Ingress控制器作为流量调度枢纽。去年我在某电商平台的迁移项目中就遇到过惨痛教训虽然集群本身配置了多Master节点但由于Nginx Ingress采用单副本部署当该节点突发宕机时整个平台的对外API服务中断了近40分钟。这个案例让我深刻认识到完整的K8s高可用方案必须包含以下三个层级控制平面高可用通过多个Master节点组成etcd集群避免单点故障工作节点高可用合理设置Pod反亲和性确保关键业务分散在不同物理节点流量入口高可用Ingress控制器需要实现多副本部署自动故障转移本次部署选择Nginx Ingress Controller作为实现方案主要基于以下考量社区活跃度高与Kubernetes版本兼容性好支持蓝绿部署、灰度发布等高级流量管理功能性能优异单个实例可处理每秒5000请求配置灵活可通过Annotations实现精细化控制2. 前置环境检查与组件规划2.1 集群健康状态验证在开始部署Ingress前我们需要确认基础集群的运行状态符合预期。执行以下命令进行验证# 检查节点Ready状态 kubectl get nodes -o wide # 检查核心组件健康状况 kubectl get cs # 验证DNS服务是否正常 kubectl run -it --rm --restartNever dig-test --imageinfoblox/dnstools:latest -- dig kubernetes.default.svc.cluster.local预期输出应显示所有节点状态为Ready核心组件Healthy且能够正常解析集群内域名。如果发现异常需要优先排查网络插件如Calico或DNS服务CoreDNS的配置问题。2.2 网络拓扑规划建议合理的网络规划能有效避免后续的性能瓶颈。根据生产环境经验我推荐以下部署模式组件副本数部署方式节点选择建议Ingress Controller3DaemonSet专属Ingress节点组CoreDNS3Deployment分散在不同可用区业务应用2DeploymentPodAntiAffinity与Ingress节点隔离这种架构设计可以带来三个优势DaemonSet确保每个物理节点运行一个Ingress实例避免端口冲突专用节点组避免业务Pod与流量转发组件资源竞争反亲和性策略降低单点故障风险3. Ingress控制器部署实战3.1 使用Helm部署Nginx Ingress官方推荐的安装方式是使用Helm Chart这比直接apply manifest文件更便于后续升级和维护。以下是具体操作步骤# 添加Ingress-Nginx仓库 helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx helm repo update # 定制化values配置文件 cat ingress-values.yaml EOF controller: replicaCount: 3 hostNetwork: true kind: DaemonSet nodeSelector: node-role.kubernetes.io/ingress: true tolerations: - key: node-role.kubernetes.io/ingress operator: Exists effect: NoSchedule service: type: ClusterIP metrics: enabled: true serviceMonitor: enabled: true EOF # 安装Ingress-Nginx helm install ingress-nginx ingress-nginx/ingress-nginx \ -n ingress-nginx --create-namespace \ -f ingress-values.yaml关键参数解析hostNetwork: true直接使用主机网络提升网络性能kind: DaemonSet确保每个节点运行一个PodnodeSelector限制只在标记为ingress的节点上运行metrics.enabled开启Prometheus指标暴露3.2 节点标记与网络配置为专用节点打上标签并配置内核参数优化网络性能# 标记Ingress专用节点 kubectl label nodes node1 node2 node3 node-role.kubernetes.io/ingresstrue # 调整内核参数 echo net.core.somaxconn 32768 net.ipv4.tcp_max_syn_backlog 8192 net.ipv4.tcp_tw_reuse 1 fs.file-max 2097152 | sudo tee /etc/sysctl.d/99-ingress.conf # 应用参数修改 sudo sysctl -p /etc/sysctl.d/99-ingress.conf重要提示如果节点已经运行其他Pod需要先排空节点kubectl drain node-name --ignore-daemonsets --delete-emptydir-data4. 高可用业务部署实践4.1 示例应用部署与Ingress配置下面通过一个实际案例演示如何部署高可用业务。我们以Nginx作为示例应用# nginx-deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-demo spec: replicas: 3 selector: matchLabels: app: nginx-demo template: metadata: labels: app: nginx-demo spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - nginx-demo topologyKey: kubernetes.io/hostname containers: - name: nginx image: nginx:1.25 ports: - containerPort: 80 --- # nginx-ingress.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: nginx-demo annotations: nginx.ingress.kubernetes.io/affinity: cookie nginx.ingress.kubernetes.io/affinity-mode: persistent spec: ingressClassName: nginx rules: - host: demo.example.com http: paths: - path: / pathType: Prefix backend: service: name: nginx-demo port: number: 80关键配置说明podAntiAffinity确保Pod分散在不同节点affinity注解实现会话保持ingressClassName指定使用的Ingress控制器4.2 高级流量管理技巧在生产环境中我们通常需要更精细的流量控制蓝绿发布annotations: nginx.ingress.kubernetes.io/canary: true nginx.ingress.kubernetes.io/canary-weight: 20基于Header的路由annotations: nginx.ingress.kubernetes.io/canary-by-header: X-Env nginx.ingress.kubernetes.io/canary-by-header-value: staging限流保护annotations: nginx.ingress.kubernetes.io/limit-rpm: 100 nginx.ingress.kubernetes.io/limit-burst: 505. 监控与自动化运维5.1 Prometheus监控集成Ingress控制器的监控指标对于容量规划至关重要。以下是集成Prometheus的配置示例# ingress-monitor.yaml apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: ingress-nginx labels: release: prometheus spec: selector: matchLabels: app.kubernetes.io/instance: ingress-nginx endpoints: - port: metrics interval: 15s path: /metrics关键指标告警规则建议nginx_ingress_controller_requests请求量突增/突降nginx_ingress_controller_nginx_process_requests_total5xx错误率nginx_ingress_controller_connections连接数接近上限5.2 自动化扩缩容配置结合HPA实现Ingress控制器自动扩缩容apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: ingress-nginx spec: scaleTargetRef: apiVersion: apps/v1 kind: DaemonSet name: ingress-nginx-controller minReplicas: 3 maxReplicas: 10 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 70 - type: External external: metric: name: nginx_ingress_controller_requests_per_second selector: matchLabels: app: ingress-nginx target: type: AverageValue averageValue: 10006. 故障排查与性能优化6.1 常见问题速查表故障现象可能原因解决方案502 Bad Gateway后端服务不可用检查Service selector与Pod标签是否匹配413 Request Entity Too Large客户端上传大文件添加注解nginx.ingress.kubernetes.io/proxy-body-size: 20m连接超时节点网络插件问题检查Calico/IPVS配置验证节点间网络连通性证书过期Lets Encrypt证书未自动续期检查cert-manager日志验证ClusterIssuer配置6.2 性能调优实战经验经过多次压力测试我总结了以下性能优化组合内核参数调优# 增加连接跟踪表大小 echo 262144 /proc/sys/net/nf_conntrack_max # 提高本地端口范围 echo 1024 65535 /proc/sys/net/ipv4/ip_local_port_rangeIngress配置优化annotations: nginx.ingress.kubernetes.io/upstream-keepalive-connections: 100 nginx.ingress.kubernetes.io/upstream-keepalive-timeout: 60 nginx.ingress.kubernetes.io/proxy-buffer-size: 16k资源限制建议resources: limits: cpu: 2 memory: 2Gi requests: cpu: 500m memory: 512Mi在电商大促期间这套配置成功支撑了单Ingress实例每秒8000请求的流量峰值。关键是要根据实际监控数据持续调整参数特别是upstream-keepalive-connections和proxy-buffer-size这两个对性能影响最大的参数。
双梁正问题 【 torch 实战】（Python代码实现）)
