当前位置：首页 > news >正文

Rabbitmq+STS+discovery_k8s +localpv部署排坑详解

news 2025/8/15 7:51:46

#作者：朱雷

文章目录

一、部署排坑
- 1.1. configmap配置文件
- 1.2. pv文件
- 1.3. sc文件
- 1.4. serviceAccount文件
- 1.5. headless-service文件
- 1.6. sts文件
二、RabbitMQ集群部署关键问题总结

一、部署排坑

1.1. configmap配置文件

编辑cm.yaml 文件

apiVersion: v1
kind: ConfigMap
metadata:name: rabbitmq-confignamespace: rabbitmq-clu-9
data:rabbitmq.conf: |# 基础配置listeners.tcp.default = 5672# management.listener.port = 15672# management.listener.ssl = falsedisk_free_limit.absolute = 1GBcluster_formation.peer_discovery_backend = k8s  #指定集群发现通过k8s插件# cluster_formation.peer_discovery_backend = rabbit_peer_discovery_k8scluster_formation.k8s.host = kubernetes.default.svc.cluster.local
cluster_formation.k8s.address_type = hostname
# 后缀中rabbitmq-clu-9 为namespace 名称保持和sts 文件中的namespace一致cluster_formation.k8s.hostname_suffix = .rabbitmq-headless.rabbitmq-clu-9.svc.cluster.localcluster_formation.discovery_retry_limit = 10
cluster_formation.discovery_retry_interval = 3000
# service_name与headless 中保持一致cluster_formation.k8s.service_name = rabbitmq-headlesscluster_formation.node_cleanup.interval = 30cluster_formation.node_cleanup.only_log_warning = falsecluster_formation.etcd.ssl_options.verify = verify_none# 内存配置vm_memory_high_watermark.relative = 0.6vm_memory_high_watermark_paging_ratio = 0.5# 日志配置log.console = truelog.console.level = debuglog.file = false# 临时启用调试日志log.connection.level = debuglog.channel.level = debuglog.queue.level = debug

1.2. pv文件

apiVersion: v1
kind: PersistentVolume
metadata:name: rabbitmq-cluster-pv-0
spec:capacity:storage: 1GiaccessModes:- ReadWriteOncestorageClassName: hostpath-storagehostPath:path: /tmp/rabbitmq/0type: DirectoryOrCreatenodeAffinity:required:nodeSelectorTerms:- matchExpressions:- key: kubernetes.io/hostnameoperator: Invalues:- 192.168.88.201   #修改为绑定的node的hostname
---
apiVersion: v1
kind: PersistentVolume
metadata:name: rabbitmq-cluster-pv-1
spec:capacity:storage: 1GiaccessModes:- ReadWriteOncestorageClassName: hostpath-storagehostPath:path: /tmp/rabbitmq/1type: DirectoryOrCreatenodeAffinity:required:nodeSelectorTerms:- matchExpressions:- key: kubernetes.io/hostnameoperator: Invalues:- 192.168.88.202    #修改为绑定的node的hostname
---
apiVersion: v1
kind: PersistentVolume
metadata:name: rabbitmq-cluster-pv-2
spec:capacity:storage: 1GiaccessModes:- ReadWriteOncestorageClassName: hostpath-storagehostPath:path: /tmp/rabbitmq/2type: DirectoryOrCreatenodeAffinity:required:nodeSelectorTerms:- matchExpressions:- key: kubernetes.io/hostnameoperator: Invalues:- 192.168.88.203   #修改为绑定的node的hostname

坑1：node选择器绑定时使用的是集群node 的hostname，如node 的IP 和Hostname 不一致，填写IP 会导致pod 为运行pending状态。

1.3. sc文件

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:name: hostpath-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

1.4. serviceAccount文件

apiVersion: v1
kind: ServiceAccount
metadata:name: rabbitmqnamespace: rabbitmq-clu-9
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:name: rabbitmq-peer-discovery
rules:
- apiGroups: [""]resources: ["nodes", "pods", "endpoints"]verbs: ["get", "list", "watch"]
- apiGroups: ["discovery.k8s.io"]resources: ["endpointslices"] verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:name: rabbitmq-peer-discovery
subjects:
- kind: ServiceAccountname: rabbitmqnamespace: rabbitmq-clu-9
roleRef:kind: ClusterRolename: rabbitmq-peer-discoveryapiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:name: rabbitmq-configmapnamespace: rabbitmq-clu-9
rules:
- apiGroups: [""]resources: ["configmaps"]verbs: ["get", "update"]resourceNames: ["rabbitmq-config"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:name: rabbitmq-configmapnamespace: rabbitmq-clu-9
subjects:
- kind: ServiceAccountname: rabbitmq
roleRef:kind: Rolename: rabbitmq-configmapapiGroup: rbac.authorization.k8s.io

坑2：在启动pod 的过程中如果集群角色rabbitmq-peer-discovery未授权nodes资源则pod一直报错启动失败

1.5. headless-service文件

apiVersion: v1
kind: Service
metadata:name: rabbitmq-headlesslabels:app: rabbitmq
spec:clusterIP: Noneports:- name: amqpport: 5672targetPort: 5672- name: managementport: 15672 targetPort: 15672- name: epmdport: 4369targetPort: 4369- name: distport: 25672targetPort: 25672selector:app: rabbitmqpublishNotReadyAddresses: true

坑3：上面这几个端口都需要暴漏出来，否则集群创建失败

1.6. sts文件

apiVersion: apps/v1
kind: StatefulSet
metadata:name: rabbitmqnamespace: rabbitmq-clu-9labels:app: rabbitmq
spec:serviceName: rabbitmq-headlessreplicas: 3#podManagementPolicy: "Parallel"selector:matchLabels:app: rabbitmqtemplate:metadata:labels:app: rabbitmqspec:terminationGracePeriodSeconds: 10affinity:podAntiAffinity:preferredDuringSchedulingIgnoredDuringExecution:- weight: 100podAffinityTerm:labelSelector:matchExpressions:- key: appoperator: Invalues:- rabbitmqtopologyKey: kubernetes.io/hostnameserviceAccountName: rabbitmq        containers:- name: rabbitmqimage: rabbitmq:3.8.27-managementimagePullPolicy: IfNotPresentports:- containerPort: 5672name: amqp- containerPort: 15672name: httpenv:- name: RABBITMQ_USE_LONGNAMEvalue: "true"        - name: RABBITMQ_ERLANG_COOKIEvalue: "secret-cookie"- name: RABBITMQ_DEFAULT_USERvalue: "admin"- name: RABBITMQ_DEFAULT_PASSvalue: "admin123"volumeMounts:- name: configmountPath: /etc/rabbitmq/rabbitmq.confsubPath: rabbitmq.conf- name: datamountPath: /var/lib/rabbitmqreadOnly: false#readinessProbe:#  exec:#    command: ["rabbitmq-diagnostics", "status"]#  initialDelaySeconds: 20#  periodSeconds: 30livenessProbe:exec:command: ["rabbitmq-diagnostics", "ping"]initialDelaySeconds: 60periodSeconds: 30volumes:- name: configconfigMap:name: rabbitmq-configvolumeClaimTemplates:- metadata:name: datanamespace: rabbitmq-clu-9spec:accessModes: [ "ReadWriteOnce" ]resources:requests:storage: 1000MstorageClassName: hostpath-storage

坑4：RABBITMQ_USE_LONGNAME 环境变量需要指定为true，指定使用FQDN格式避免截全节点通信失败，集群建立失败。

坑5：如果podManagementPolicy 策略不为 “Parallel”，则readiness 探针需要关闭，避免集群启动失败。