Rabbitmq instances are crashing - linux

Question

I am using docker-compose.yml file to spin 3 instances of RabbitMQ in a single host. Running docker on mac, When I run docker-compose up, I see erlang cookies are not matching for the instances in the cluster. Let me know if you need any other information.
version: '3'
services:
rabbitmq1:
image: rabbitmq:3.8.34-management
hostname: rabbitmq1
environment:
- RABBITMQ_ERLANG_COOKIE=12345
- RABBITMQ_DEFAULT_USER=guest
- RABBITMQ_DEFAULT_PASS=guest
- RABBITMQ_DEFAULT_VHOST=/
rabbitmq2:
image: rabbitmq:3.8.34-management
hostname: rabbitmq2
depends_on:
- rabbitmq1
environment:
- RABBITMQ_ERLANG_COOKIE=12345
- RABBITMQ_DEFAULT_USER=guest
- RABBITMQ_DEFAULT_PASS=guest
- RABBITMQ_DEFAULT_VHOST=/
volumes:
- ./cluster-entrypoint.sh:/usr/local/bin/cluster-entrypoint.sh
entrypoint: /usr/local/bin/cluster-entrypoint.sh
rabbitmq3:
image: rabbitmq:3.8.34-management
hostname: rabbitmq3
depends_on:
- rabbitmq1
environment:
- RABBITMQ_ERLANG_COOKIE=12345
- RABBITMQ_DEFAULT_USER=guest
- RABBITMQ_DEFAULT_PASS=guest
- RABBITMQ_DEFAULT_VHOST=/
volumes:
- ./cluster-entrypoint.sh:/usr/local/bin/cluster-entrypoint.sh
entrypoint: /usr/local/bin/cluster-entrypoint.sh
haproxy:
image: haproxy:1.7
volumes:
- ./haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
depends_on:
- rabbitmq1
- rabbitmq2
- rabbitmq3
ports:
- 15672:15672
- 5672:5672
Below is my cluster-entrypoint.sh file
#!/bin/bash
set -e
# Start RMQ from entry point.
# This will ensure that environment variables passed
# will be honored
/usr/local/bin/docker-entrypoint.sh rabbitmq-server -detached
# Do the cluster dance
rabbitmqctl stop_app
rabbitmqctl join_cluster rabbit#rabbitmq1
# Stop the entire RMQ server. This is done so that we
# can attach to it again, but without the -detached flag
# making it run in the forground
rabbitmqctl stop
# Wait a while for the app to really stop
sleep 2s
# Start it
rabbitmq-server
Sorry for too many logs. I have used RabbitMQ:3.8.34 image, erlang cookies for all instances in cluster are different and RabbitMQ1 starts but other instances does not start.
Below is the log:
haproxy_1 | <7>haproxy-systemd-wrapper: executing /usr/local/sbin/haproxy -p /run/haproxy.pid -db -f /usr/local/etc/haproxy/haproxy.cfg -Ds
rabbitmq1_1 |
rabbitmq1_1 | warning: /var/lib/rabbitmq/.erlang.cookie contents do not match RABBITMQ_ERLANG_COOKIE
rabbitmq1_1 |
rabbitmq1_1 | WARNING: '/var/lib/rabbitmq/.erlang.cookie' was populated from '$RABBITMQ_ERLANG_COOKIE', which will no longer happen in 3.9 and later! (https://github.com/docker-library/rabbitmq/pull/424)
rabbitmq2_1 |
rabbitmq2_1 | warning: /var/lib/rabbitmq/.erlang.cookie contents do not match RABBITMQ_ERLANG_COOKIE
rabbitmq2_1 |
rabbitmq2_1 | WARNING: '/var/lib/rabbitmq/.erlang.cookie' was populated from '$RABBITMQ_ERLANG_COOKIE', which will no longer happen in 3.9 and later! (https://github.com/docker-library/rabbitmq/pull/424)
rabbitmq3_1 |
rabbitmq3_1 | warning: /var/lib/rabbitmq/.erlang.cookie contents do not match RABBITMQ_ERLANG_COOKIE
rabbitmq3_1 |
rabbitmq3_1 | WARNING: '/var/lib/rabbitmq/.erlang.cookie' was populated from '$RABBITMQ_ERLANG_COOKIE', which will no longer happen in 3.9 and later! (https://github.com/docker-library/rabbitmq/pull/424)
rabbitmq1_1 | WARNING: 'docker-entrypoint.sh' generated/modified the RabbitMQ configuration file, which will no longer happen in 3.9 and later! (https://github.com/docker-library/rabbitmq/pull/424)
rabbitmq1_1 |
rabbitmq1_1 | Generated end result, for reference:
rabbitmq1_1 | ------------------------------------
rabbitmq1_1 | loopback_users.guest = false
rabbitmq1_1 | listeners.tcp.default = 5672
rabbitmq1_1 | default_pass = guest
rabbitmq1_1 | default_user = guest
rabbitmq1_1 | default_vhost = /
rabbitmq1_1 | management.tcp.port = 15672
rabbitmq1_1 | ------------------------------------
rabbitmq3_1 | WARNING: 'docker-entrypoint.sh' generated/modified the RabbitMQ configuration file, which will no longer happen in 3.9 and later! (https://github.com/docker-library/rabbitmq/pull/424)
rabbitmq3_1 |
rabbitmq3_1 | Generated end result, for reference:
rabbitmq3_1 | ------------------------------------
rabbitmq3_1 | loopback_users.guest = false
rabbitmq3_1 | listeners.tcp.default = 5672
rabbitmq3_1 | default_pass = guest
rabbitmq3_1 | default_user = guest
rabbitmq3_1 | default_vhost = /
rabbitmq3_1 | management.tcp.port = 15672
rabbitmq3_1 | ------------------------------------
rabbitmq2_1 | WARNING: 'docker-entrypoint.sh' generated/modified the RabbitMQ configuration file, which will no longer happen in 3.9 and later! (https://github.com/docker-library/rabbitmq/pull/424)
rabbitmq2_1 |
rabbitmq2_1 | Generated end result, for reference:
rabbitmq2_1 | ------------------------------------
rabbitmq2_1 | loopback_users.guest = false
rabbitmq2_1 | listeners.tcp.default = 5672
rabbitmq2_1 | default_pass = guest
rabbitmq2_1 | default_user = guest
rabbitmq2_1 | default_vhost = /
rabbitmq2_1 | management.tcp.port = 15672
rabbitmq2_1 | ------------------------------------
rabbitmq3_1 | RABBITMQ_ERLANG_COOKIE env variable support is deprecated and will be REMOVED in a future version. Use the $HOME/.erlang.cookie file or the --erlang-cookie switch instead.
rabbitmq3_1 | Stopping rabbit application on node rabbit#rabbitmq3 ...
rabbitmq3_1 | Error: unable to perform an operation on node 'rabbit#rabbitmq3'. Please see diagnostics information and suggestions below.
rabbitmq3_1 |
rabbitmq3_1 | Most common reasons for this are:
rabbitmq3_1 |
rabbitmq3_1 | * Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
rabbitmq3_1 | * CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)
rabbitmq3_1 | * Target node is not running
rabbitmq3_1 |
rabbitmq3_1 | In addition to the diagnostics info below:
rabbitmq3_1 |
rabbitmq3_1 | * See the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more
rabbitmq3_1 | * Consult server logs on node rabbit#rabbitmq3
rabbitmq3_1 | * If target node is configured to use long node names, don't forget to use --longnames with CLI tools
rabbitmq3_1 |
rabbitmq3_1 | DIAGNOSTICS
rabbitmq3_1 | ===========
rabbitmq3_1 |
rabbitmq3_1 | attempted to contact: [rabbit#rabbitmq3]
rabbitmq3_1 |
rabbitmq3_1 | rabbit#rabbitmq3:
rabbitmq3_1 | * connected to epmd (port 4369) on rabbitmq3
rabbitmq3_1 | * epmd reports: node 'rabbit' not running at all
rabbitmq3_1 | no other nodes on rabbitmq3
rabbitmq3_1 | * suggestion: start the node
rabbitmq3_1 |
rabbitmq3_1 | Current node details:
rabbitmq3_1 | * node name: 'rabbitmqcli-797-rabbit#rabbitmq3'
rabbitmq3_1 | * effective user's home directory: /var/lib/rabbitmq
rabbitmq3_1 | * Erlang cookie hash: gnzLDuqKcGxMNKFokfhOew==
rabbitmq3_1 |
docker-rabbitmq-cluster_rabbitmq3_1 exited with code 69
rabbitmq2_1 | RABBITMQ_ERLANG_COOKIE env variable support is deprecated and will be REMOVED in a future version. Use the $HOME/.erlang.cookie file or the --erlang-cookie switch instead.
rabbitmq2_1 | Stopping rabbit application on node rabbit#rabbitmq2 ...
rabbitmq2_1 | Error: unable to perform an operation on node 'rabbit#rabbitmq2'. Please see diagnostics information and suggestions below.
rabbitmq2_1 |
rabbitmq2_1 | Most common reasons for this are:
rabbitmq2_1 |
rabbitmq2_1 | * Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
rabbitmq2_1 | * CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)
rabbitmq2_1 | * Target node is not running
rabbitmq2_1 |
rabbitmq2_1 | In addition to the diagnostics info below:
rabbitmq2_1 |
rabbitmq2_1 | * See the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more
rabbitmq2_1 | * Consult server logs on node rabbit#rabbitmq2
rabbitmq2_1 | * If target node is configured to use long node names, don't forget to use --longnames with CLI tools
rabbitmq2_1 |
rabbitmq2_1 | DIAGNOSTICS
rabbitmq2_1 | ===========
rabbitmq2_1 |
rabbitmq2_1 | attempted to contact: [rabbit#rabbitmq2]
rabbitmq2_1 |
rabbitmq2_1 | rabbit#rabbitmq2:
rabbitmq2_1 | * connected to epmd (port 4369) on rabbitmq2
rabbitmq2_1 | * epmd reports: node 'rabbit' not running at all
rabbitmq2_1 | no other nodes on rabbitmq2
rabbitmq2_1 | * suggestion: start the node
rabbitmq2_1 |
rabbitmq2_1 | Current node details:
rabbitmq2_1 | * node name: 'rabbitmqcli-568-rabbit#rabbitmq2'
rabbitmq2_1 | * effective user's home directory: /var/lib/rabbitmq
rabbitmq2_1 | * Erlang cookie hash: gnzLDuqKcGxMNKFokfhOew==
rabbitmq2_1 |
docker-rabbitmq-cluster_rabbitmq2_1 exited with code 69
rabbitmq1_1 | Configuring logger redirection
rabbitmq1_1 | 2022-07-05 02:43:40.659 [debug] <0.288.0> Lager installed handler error_logger_lager_h into error_logger
rabbitmq1_1 | 2022-07-05 02:43:40.670 [debug] <0.291.0> Lager installed handler lager_forwarder_backend into error_logger_lager_event
rabbitmq1_1 | 2022-07-05 02:43:40.670 [debug] <0.312.0> Lager installed handler lager_forwarder_backend into rabbit_log_mirroring_lager_event
rabbitmq1_1 | 2022-07-05 02:43:40.670 [debug] <0.303.0> Lager installed handler lager_forwarder_backend into rabbit_log_feature_flags_lager_event
rabbitmq1_1 | 2022-07-05 02:43:40.670 [debug] <0.309.0> Lager installed handler lager_forwarder_backend into rabbit_log_ldap_lager_event
rabbitmq1_1 | 2022-07-05 02:43:40.670 [debug] <0.294.0> Lager installed handler lager_forwarder_backend into rabbit_log_lager_event
rabbitmq1_1 | 2022-07-05 02:43:40.670 [debug] <0.297.0> Lager installed handler lager_forwarder_backend into rabbit_log_channel_lager_event
rabbitmq1_1 | 2022-07-05 02:43:40.670 [debug] <0.306.0> Lager installed handler lager_forwarder_backend into rabbit_log_federation_lager_event
rabbitmq1_1 | 2022-07-05 02:43:40.670 [debug] <0.300.0> Lager installed handler lager_forwarder_backend into rabbit_log_connection_lager_event
rabbitmq1_1 | 2022-07-05 02:43:40.671 [debug] <0.315.0> Lager installed handler lager_forwarder_backend into rabbit_log_prelaunch_lager_event
rabbitmq1_1 | 2022-07-05 02:43:40.672 [debug] <0.318.0> Lager installed handler lager_forwarder_backend into rabbit_log_queue_lager_event
rabbitmq1_1 | 2022-07-05 02:43:40.673 [debug] <0.321.0> Lager installed handler lager_forwarder_backend into rabbit_log_ra_lager_event
rabbitmq1_1 | 2022-07-05 02:43:40.675 [debug] <0.324.0> Lager installed handler lager_forwarder_backend into rabbit_log_shovel_lager_event
rabbitmq1_1 | 2022-07-05 02:43:40.676 [debug] <0.327.0> Lager installed handler lager_forwarder_backend into rabbit_log_upgrade_lager_event
rabbitmq1_1 | 2022-07-05 02:43:40.691 [info] <0.44.0> Application lager started on node rabbit#rabbitmq1
rabbitmq1_1 | 2022-07-05 02:43:41.159 [debug] <0.284.0> Lager installed handler lager_backend_throttle into lager_event
haproxy_1 | [ALERT] 185/024336 (8) : parsing [/usr/local/etc/haproxy/haproxy.cfg:32] : 'server rabbitmq2' : could not resolve address 'rabbitmq2'.
haproxy_1 | [ALERT] 185/024336 (8) : parsing [/usr/local/etc/haproxy/haproxy.cfg:33] : 'server rabbitmq3' : could not resolve address 'rabbitmq3'.
haproxy_1 | [ALERT] 185/024336 (8) : parsing [/usr/local/etc/haproxy/haproxy.cfg:43] : 'server rabbitmq2' : could not resolve address 'rabbitmq2'.
haproxy_1 | [ALERT] 185/024336 (8) : parsing [/usr/local/etc/haproxy/haproxy.cfg:44] : 'server rabbitmq3' : could not resolve address 'rabbitmq3'.
haproxy_1 | [ALERT] 185/024336 (8) : Failed to initialize server(s) addr.
haproxy_1 | <5>haproxy-systemd-wrapper: exit, haproxy RC=1
docker-rabbitmq-cluster_haproxy_1 exited with code 1
rabbitmq1_1 | 2022-07-05 02:43:43.065 [info] <0.44.0> Application mnesia started on node rabbit#rabbitmq1
rabbitmq1_1 | 2022-07-05 02:43:43.066 [info] <0.273.0>
rabbitmq1_1 | Starting RabbitMQ 3.8.34 on Erlang 24.3.4.1 [emu]
rabbitmq1_1 | Copyright (c) 2007-2022 VMware, Inc. or its affiliates.
rabbitmq1_1 | Licensed under the MPL 2.0. Website: https://rabbitmq.com
rabbitmq1_1 |
rabbitmq1_1 | ## ## RabbitMQ 3.8.34
rabbitmq1_1 | ## ##
rabbitmq1_1 | ########## Copyright (c) 2007-2022 VMware, Inc. or its affiliates.
rabbitmq1_1 | ###### ##
rabbitmq1_1 | ########## Licensed under the MPL 2.0. Website: https://rabbitmq.com
rabbitmq1_1 |
rabbitmq1_1 | Erlang: 24.3.4.1 [emu]
rabbitmq1_1 | TLS Library: OpenSSL - OpenSSL 1.1.1o 3 May 2022
rabbitmq1_1 |
rabbitmq1_1 | Doc guides: https://rabbitmq.com/documentation.html
rabbitmq1_1 | Support: https://rabbitmq.com/contact.html
rabbitmq1_1 | Tutorials: https://rabbitmq.com/getstarted.html
rabbitmq1_1 | Monitoring: https://rabbitmq.com/monitoring.html
rabbitmq1_1 |
rabbitmq1_1 | Logs: <stdout>
rabbitmq1_1 |
rabbitmq1_1 | Config file(s): /etc/rabbitmq/rabbitmq.conf
rabbitmq1_1 |
rabbitmq1_1 | Starting broker...2022-07-05 02:43:43.068 [info] <0.273.0>
rabbitmq1_1 | node : rabbit#rabbitmq1
rabbitmq1_1 | home dir : /var/lib/rabbitmq
rabbitmq1_1 | config file(s) : /etc/rabbitmq/rabbitmq.conf
rabbitmq1_1 | cookie hash : VlfoFK5J8f9Ln3G9sXDoPQ==
rabbitmq1_1 | log(s) : <stdout>
rabbitmq1_1 | database dir : /var/lib/rabbitmq/mnesia/rabbit#rabbitmq1
rabbitmq1_1 | 2022-07-05 02:43:44.265 [info] <0.44.0> Application amqp_client started on node rabbit#rabbitmq1
rabbitmq1_1 | 2022-07-05 02:43:44.279 [info] <0.584.0> Management plugin: HTTP (non-TLS) listener started on port 15672
rabbitmq1_1 | 2022-07-05 02:43:44.279 [info] <0.612.0> Statistics database started.
rabbitmq1_1 | 2022-07-05 02:43:44.279 [info] <0.611.0> Starting worker pool 'management_worker_pool' with 3 processes in it
rabbitmq1_1 | 2022-07-05 02:43:44.279 [info] <0.44.0> Application rabbitmq_management started on node rabbit#rabbitmq1
rabbitmq1_1 | 2022-07-05 02:43:44.292 [info] <0.44.0> Application prometheus started on node rabbit#rabbitmq1
rabbitmq1_1 | 2022-07-05 02:43:44.294 [info] <0.625.0> Prometheus metrics: HTTP (non-TLS) listener started on port 15692
rabbitmq1_1 | 2022-07-05 02:43:44.294 [info] <0.525.0> Ready to start client connection listeners
rabbitmq1_1 | 2022-07-05 02:43:44.294 [info] <0.44.0> Application rabbitmq_prometheus started on node rabbit#rabbitmq1
rabbitmq1_1 | 2022-07-05 02:43:44.297 [info] <0.669.0> started TCP listener on [::]:5672
rabbitmq1_1 | 2022-07-05 02:43:45.321 [info] <0.525.0> Server startup complete; 4 plugins started.
rabbitmq1_1 | * rabbitmq_prometheus
rabbitmq1_1 | * rabbitmq_management
rabbitmq1_1 | * rabbitmq_web_dispatch
rabbitmq1_1 | * rabbitmq_management_agent
rabbitmq1_1 | completed with 4 plugins.
rabbitmq1_1 | 2022-07-05 02:43:45.322 [info] <0.525.0> Resetting node maintenance status
I am not sure what I am missing. Sorry for too much of logs.

Answer 1

After analysing your files and log files I can observe few issues.
Your setup will be able to start only once while there no files/configuration and you are doing it from the scratch. The reason of such behaviour is that RabbitMQ stores configuration to internal DB called Mnesia and it is mandatory that once nodes added it should be present for further start. If you won't follow it you will observer errors that the node waiting for mnesia to find its nodes.
Another issue with repetitive start is that node that was added (2 or 3) to a cluster marks itself as a cluster member, you will see an error that main node (1) expects to connect to earlier connected but your entry-point already reset 2 and 3 nodes.
You cannot combine versions of RabbitMQ cause it might contain different data structures that will not allow nodes to sync and you will have an error like "schema_integrity_check_failed...", it should be all identical.
When I was using RabbitMQ, my configuration ensures that there is persistent location (disk) with all the data and that repetitive start will use already initialised data. Also it is a good practice to use cluster management software such as etc or consul that is supported by RabbitMQ and you don't need to handle it yourself.
Hope that would help.
Generally I was able to successfully start your setup on my machine (macOS) with docker, the script is the following:
ensure you never started the compose in order to avoid the data from previous runs
prepare everything in the folder
do docker-compose up, enjoy everything works
use docker-compose down for cleanup the stack not '... stop' command cause the data will stay
Below you can find a log for the start, please ignore ha_proxy error, there is no config file attached so it must fail
% docker-compose up
Creating network "rabbitmq_default" with the default driver
Creating rabbitmq_rabbitmq1_1 ... done
Creating rabbitmq_rabbitmq3_1 ... done
Creating rabbitmq_rabbitmq2_1 ... done
Creating rabbitmq_haproxy_1 ... done
Attaching to rabbitmq_rabbitmq1_1, rabbitmq_rabbitmq3_1, rabbitmq_rabbitmq2_1, rabbitmq_haproxy_1
(due to limitation I've put it here)
My setup looks like this, the only change is that rabbitMQ data is persisted and init is done manually, you also can mount data folder from file system path
version: '3'
services:
rabbitmq1:
image: rabbitmq:3.8.34-management
hostname: rabbitmq1
environment:
- RABBITMQ_ERLANG_COOKIE=12345
- RABBITMQ_DEFAULT_USER=guest
- RABBITMQ_DEFAULT_PASS=guest
- RABBITMQ_DEFAULT_VHOST=/
volumes:
- rabbitmq-01-data:/var/lib/rabbitmq
rabbitmq2:
image: rabbitmq:3.8.34-management
hostname: rabbitmq2
depends_on:
- rabbitmq1
environment:
- RABBITMQ_ERLANG_COOKIE=12345
- RABBITMQ_DEFAULT_USER=guest
- RABBITMQ_DEFAULT_PASS=guest
- RABBITMQ_DEFAULT_VHOST=/
volumes:
- rabbitmq-02-data:/var/lib/rabbitmq
rabbitmq3:
image: rabbitmq:3.8.34-management
hostname: rabbitmq3
depends_on:
- rabbitmq1
environment:
- RABBITMQ_ERLANG_COOKIE=12345
- RABBITMQ_DEFAULT_USER=guest
- RABBITMQ_DEFAULT_PASS=guest
- RABBITMQ_DEFAULT_VHOST=/
volumes:
- rabbitmq-03-data:/var/lib/rabbitmq
volumes:
rabbitmq-01-data:
rabbitmq-02-data:
rabbitmq-03-data:
Manual run on every "follower" node
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl join_cluster rabbit#rabbitmq1
rabbitmqctl start_app
Don't forget to enable redundant queues if applicable for your case
rabbitmqctl set_policy ha "." '{"ha-mode":"all"}'

Answer 2

I have a complete RabbitMQ setup that forms a cluster using docker-compose here:
https://github.com/lukebakken/docker-rabbitmq-cluster
Please note that the following mirroring policy is NOT recommended. There is no need to mirror queues to all nodes -
rabbitmqctl set_policy ha "." '{"ha-mode":"all"}'
You should mirror to 2 nodes in your cluster.
BETTER YET, use the latest version of RabbitMQ and use Quorum Queues! Classic HA mirroring will be removed from RabbitMQ in version 4.0
https://www.rabbitmq.com/quorum-queues.html
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.