How to do Rabbitmq Clustering

Setting up Rabbitmq cluster of 2 nodes

Step-0: Verify version on all nodes

For clustering all nodes have to have the same version, you can check the version by the command below.

sudo rabbitmqctl status

Look for 

{rabbit,"RabbitMQ","2.6.1"},

Step-1: Setup Cookies

All the nodes in a cluster need to have same content in a cookie file. Just simply copy paste the value from 1 server and paste into other servers. New servers which wish to join a cluster need to do this as well. Cookie file is usually placed at this location: /var/lib/rabbitmq/.erlang.cookie 

Step-1.5: Setup Management Plugin(Optional)

Install the management plugin to configure rabbitmq from your browser. Very helpful!

rabbitmq-plugins enable rabbitmq_management

Step-2: Setup Clustering

Either node can independently join the other and the recipient will automatically know that the node is part of the cluster now. So it’s simple to add new nodes as there is no change in existing running nodes.

Suppose we have 2 nodes running with rabbitmq installed

Node1: Rock-Mob-CC-DB01

Node2: Rock-Mob-CC-DB02

In this case Rock-Mob-CC-DB02 is joining Rock-Mob-CC-DB01. The following commands will need to be run on Rock-Mob-CC-DB02

1 Rock-Mob-CC-DB02: rabbitmqctl stop_app

2 Rock-Mob-CC-DB02: rabbitmqctl join_cluster rabbit@Rock-Mob-CC-DB01

3 Rock-Mob-CC-DB02: rabbitmqctl start_app

4 Rock-Mob-CC-DB02: rabbitmqctl cluster_status
Cluster status of node 'rabbit@Rock-Mob-CC-DB02' ...

[{nodes,[{disc,['rabbit@Rock-Mob-CC-DB01','rabbit@Rock-Mob-CC-DB02']}]},

{running_nodes,['rabbit@Rock-Mob-CC-DB01','rabbit@Rock-Mob-CC-DB02']},

{cluster_name,<<"rabbit@Rock-Mob-CC-DB01">>},

{partitions,[]}]

Notice the highlighted part, node Rock-Mob-CC-DB01 is part of the cluster now.

Similarly checking the status of cluster on Rock-Mob-CC-DB01

Rock-Mob-CC-DB01:~$ sudo rabbitmqctl cluster_status

Cluster status of node 'rabbit@Rock-Mob-CC-DB01' ...

[{nodes,[{disc,['rabbit@Rock-Mob-CC-DB01','rabbit@Rock-Mob-CC-DB02']}]},

 {running_nodes,['rabbit@Rock-Mob-CC-DB02','rabbit@Rock-Mob-CC-DB01']},

 {cluster_name,<<"rabbit@Rock-Mob-CC-DB01">>},

 {partitions,[]}]

We are sure now that both nodes are aware of the each other

Step-3: Achieving High Availibility

Now the servers are in a cluster but the queues will still not be available if one of the node goes down. We need to add physical mirroring now.

Mirroring Queues

Now nodes are in a cluster BUT the queues are not physically replicated. As per documentation queues within a RabbitMQ cluster are located on a single node (the node on which they were first declared). To enable physical mirroring we need to define a policy.

Example Policy:

Policy where queues whose names begin with "ha" are mirrored to all nodes in the cluster:

Navigate to Admin > Policies > Add / update a policy.

Enter "ha-all" next to Name, "^ha" next to Pattern, and "ha-mode" = "all" in the first line next to Policy and “ha-sync-mode”=”automatic”

Click Add policy.

This is it, the queues are physically replicated. When one node goes down, the other nodes will take over and data will still be present. This can be tested by running

Rock-Mob-CC-DB01: rabbitmqctl list_queues name slave_pids synchronised_slave_pids

Listing queues ...

ccq_auto_call   []     []

ccq_auto_miss_call      []     []

ccq_auto_notify_sms     []     []

ccq_init_requests       []     []

Mirroring can also be verified by physically downing a node and see the status of queue from other node.

Reference Material:

https://www.rabbitmq.com/clustering.html
https://www.rabbitmq.com/ha.html