Installing RabbitMQ & Celery

• RabbitMQ - Message broker server built on the Advanced Message Queuing Protocol (AMQP). RabbitMQ is written in Erlang. It's responsible queuing up tasks and scheduling them.
• Celery - Task queue that is built on an asynchronous message passing system. Celery is written in Python. It can be used as a wrapper for Python API to interact with RabbitMQ.
• Celeryd - Part of the Celery package and it is the worker that actually runs the task.
• Producer (Publisher) - A program that sends messages.
• Consumer - A program that mostly waits to receive messages.

RabbitMQ and Celery work together to execute a code sometime later when resources are available.

RabbitMQ install

We'll work on Ubuntu 14.04. RabbitMQ installation on Ubuntu is as simple as this:

$ sudo apt-get install rabbitmq-server

Rabbitmq is set to start automatically after it's installed.
To see the information about the installed RabbitMQ such as version:

$ sudo rabbitmqctl status
Status of node rabbit@k ...
[{pid,5135},
 {running_applications,[{rabbit,"RabbitMQ","3.2.4"},
                        {mnesia,"MNESIA  CXC 138 12","4.11"},
                        {os_mon,"CPO  CXC 138 46","2.2.14"},
                        {xmerl,"XML parser","1.3.5"},
                        {sasl,"SASL  CXC 138 11","2.3.4"},
                        {stdlib,"ERTS  CXC 138 10","1.19.4"},
                        {kernel,"ERTS  CXC 138 10","2.16.4"}]},
 {os,{unix,linux}},
 {erlang_version,"Erlang R16B03 (erts-5.10.4) [source] [64-bit] [smp:2:2] [async-threads:30] [kernel-poll:true]\n"},
 {memory,[{total,35396072},
          {connection_procs,2704},
          {queue_procs,5408},
          {plugins,0},
          {other_proc,13384560},
          {mnesia,60688},
          {mgmt_db,0},
          {msg_index,28112},
          {other_ets,761752},
          {binary,8632},
          {code,16522377},
          {atom,594537},
          {other_system,4027302}]},
 {vm_memory_high_watermark,0.4},
 {vm_memory_limit,1487224832},
 {disk_free_limit,50000000},
 {disk_free,439562670080},
 {file_descriptors,[{total_limit,924},
                    {total_used,3},
                    {sockets_limit,829},
                    {sockets_used,1}]},
 {processes,[{limit,1048576},{used,123}]},
 {run_queue,0},
 {uptime,6387}]
...done.

To stop the rabbitmq:

$ sudo rabbitmqctl stop
[sudo] password for k: 
Stopping and halting node rabbit@k ...
...done.

Issue the status command again to see that it's really stopped.
$ sudo rabbitmqctl status

Error: unable to connect to node rabbit@k: nodedown

DIAGNOSTICS
===========

nodes in question: [rabbit@k]

hosts, their running nodes and ports:
- k: [{rabbitmqctl8566,53150}]

current node details:
- node name: rabbitmqctl8566@k
- home dir: /var/lib/rabbitmq
- cookie hash: 37CWzJUtjVqK+RSWTjgfqg==

To start it again, the recommended method is

$ sudo invoke-rc.d rabbitmq-server start
 * Starting message broker rabbitmq-server  

If we check the status again:

$ sudo rabbitmqctl status
Status of node rabbit@k ...
[{pid,8683},
 {running_applications,[{rabbit,"RabbitMQ","3.2.4"},
                        {os_mon,"CPO  CXC 138 46","2.2.14"},
                        {mnesia,"MNESIA  CXC 138 12","4.11"},
                        {xmerl,"XML parser","1.3.5"},
                        {sasl,"SASL  CXC 138 11","2.3.4"},
                        {stdlib,"ERTS  CXC 138 10","1.19.4"},
                        {kernel,"ERTS  CXC 138 10","2.16.4"}]},
 {os,{unix,linux}},
 {erlang_version,"Erlang R16B03 (erts-5.10.4) [source] [64-bit] [smp:2:2] [async-threads:30] [kernel-poll:true]\n"},
 {memory,[{total,35605480},
          {connection_procs,2704},
          {queue_procs,5408},
          {plugins,0},
          {other_proc,13607584},
          {mnesia,60128},
          {mgmt_db,0},
          {msg_index,20888},
          {other_ets,755712},
          {binary,7488},
          {code,16522377},
          {atom,594537},
          {other_system,4028654}]},
 {vm_memory_high_watermark,0.4},
 {vm_memory_limit,1487224832},
 {disk_free_limit,50000000},
 {disk_free,439561547776},
 {file_descriptors,[{total_limit,924},
                    {total_used,3},
                    {sockets_limit,829},
                    {sockets_used,1}]},
 {processes,[{limit,1048576},{used,123}]},
 {run_queue,0},
 {uptime,56}]
...done.

Celery install

We may want to create a directory where we can implement our new system:

$ mkdir ~/TEST/MQ
$ cd ~/TEST/MQ

Now we install Celery:

$ sudo apt-get update
$ sudo pip install celery

$ which celery
/usr/local/bin/celery

$ celery --version
3.1.13 (Cipater)

We're going to use Celery later after we've done some practice with RabbitMQ