Introduction to Messaging Queues
Messaging Queues are used to communicate between two applications even if one of the applications needs to work asynchronously on long running jobs. Queue is a sequential line of messages or tasks that are to be processed. Messages are data which is to be transferred to get processed by the receiver of that message. So messaging queue is an asynchronous service-to-service communication, wherein producer produces the message and consumer consumes it. Each message is processed only once and they remain in the queue until they are processed or deleted.
In a messaging based system, there is a sender(publisher) who generates messages and send it to the shared memory or a Messaging Queue. The queue sends these messages to the receiver, asynchronously, to get processed. The sender of the message is known as producer in the messaging queue. The one who receives the messages is a consumer in the queue. There can be multiple consumers for a single queue. The messaging architecture can follow one of the below architecture.
1.) Writing into the shared memory by one process and reading from the shared memory by another process.
2.) Writing into the shared memory by one process and reading from it by multiple processes.
Importance of Messaging Queues
We can understand the importance of messaging queues by studying its implementation or usage. So, let us take some examples where messaging queues can be used.
Let us assume, some web service providing platform, which takes requests for providing certain services. If this platform gets more number of requests than it can’t process at the same time, then, requests will get pilled up and server will be overloaded. The load-balancing feature of messaging queue is used to solve this. The requests when comes in will be added to the queue so that it can be handled one by one by different servers according to server’s capacity. This way elasticity and scalability of handling requests can be achieved through Messaging Queue. Now, let us see one more example to understand importance of messaging queue.
Serverless and micro-service architecture are in use now a days. So, let us assume, there is one such banking micro-service system architecture (micro-service means multiple small servers, one for each module). This system contains authentication server, account creation server, credit-card server, loan and recovery, user detail and account balance server. Now, when there is any request regarding the account creation the account creation server will process the request and update the queue about a new account creation. Then all the other related servers will get the information from the queued message and process it in the background at its own time.
This way the account creation process will be faster, decoupled and asynchronous. Also it is reliable as it guarantees that the requests will be processed even if the server goes down or the front hand application goes down. Because the requests are logged in an isolated queue, so, when the servers get back alive they will process the request.
Now, we will get brief idea about some of the top messaging queues in use now-a-days.
RabbitMQ Messaging queue
RabbitMQ is one of the most popular open source message brokers. Broker here means the middleman, here, the queue itself. It’s lightweight and easy to deploy on premises and in the cloud with multiple messaging protocols. It runs on many operating systems.
- A RabbitMQ message broker can be created at Manifold via the CloudAMQP resource. CloudAMQP automates every part of the setting up, running and scaling of RabbitMQ clusters.
- It’s used when the data is less but has varied different requests.
- RabbitMQ is a distributed message queue system. Distributed because it usually runs as a cluster of nodes. Several RabbitMQ servers on a local network can be clustered together, forming a single logical broker.
- Queues are single-threaded in RabbitMQ, and one queue can be about 50 thousand messages long.
- It ensures that even in the event of hardware failure your messages are safe.
Apache Kafka is a distributed publish-subscribe messaging system. Publish-subscribe messaging system means where senders of messages, called publishers, do not program the messages to be sent directly to specific receivers, called subscribers, but instead categorize published messages into classes without knowledge of which subscribers, if any, there may be. Similarly, subscribers express interest in one or more classes and only receive messages that are of interest, without knowledge of which publishers, if any, there are.
- It is mainly for handling a high volume of data which is suitable for both offline and online message consumption.
- Kafka scales easily without downtime, in all the four dimensions, i.e. event producers, event processors, event consumers and event connectors.
- To prevent data loss, Kafka messages are persisted on disk and replicated within the cluster.
- It has good Fault Tolerance and can handle failures with the masters and databases.
- Kafka is very fast and guarantees zero downtime.
- Supports a variety of Cross Language Clients and Protocols from Java, C, C++, C#, Ruby, Perl, Python, PHP, etc.
Amazon SQS – Messaging Queue
Amazon Simple Queue Service (Amazon SQS) is a distributed message queuing service introduced by Amazon.com in late 2004.
- Unlike other technologies, users do not need to maintain their own server. Also high scalability is provided, if your volume grows up, amazon does it all, after all it’s an AWS.
- Messages can contain up to 256 KB of text in any format such as json, xml, etc. But, you can even store the contents of messages larger than 256 KB using Amazon S3.
- Amazon SQS locks your messages during processing, so that multiple producers can send and multiple consumers can receive messages at the same time.
- It is simple to use, and don’t require to set up message brokers.
SQS offers two types of message queues.
- Standard queues, that offers, maximum throughput, best-effort ordering, and at-least-once delivery.
- SQS FIFO queues, that are designed to guarantee that messages are processed exactly once, in the exact order that they are sent.
ActiveMQ is an open source protocol, written in Java, which functions as an implementation of message-oriented middleware (MOM).The ActiveMQ broker routes each message through one of two types of destinations:
- a queue, where it awaits delivery to a single consumer (in a messaging domain called point-to-point), or
- a topic, to be delivered to multiple consumers that are subscribed to that topic (in a messaging domain called publish/subscribe, or pub/sub).
ActiveMQ sends messages asynchronously, so producer’s task of composing and sending a message is disconnected from the consumer’s task of fetching it.
This type of arrangement, in which producer and consumer function independent of each other, is known as loose coupling. This gives you benefits of high throughput and flexibility.
ActiveMQ makes use of the Java Message Service (JMS) API, but, it’s possible to write ActiveMQ clients in other languages (such as Node.js, Ruby, and Python).
IBM MQ – Messaging Queue
IBM MQ messaging system is made up of one or more queue managers. Queue manager is a system service that provides a logical container for the message queue. A network of such connected queue managers supports asynchronous routing of messages between systems, where producing and consuming applications are connected to different queue managers.
It is a secured MQ, with Advanced Message Security (AMS), you can encrypt and sign messages between being put by one application and retrieved by another.
Message loss or duplication of messages is prevented due to its once and once only delivery mechanism.
IBM MQ Explorer GUI is also available for ease of use to Windows and Linux OS users.
It supports many different programming languages like C, Cobol, Java, .Net, NodeJS and Rubby.
Messaging queues are an important tool for executing asynchronous actions or jobs in a microservice architecture. Messaging queues acts as a buffer between the publishers and consumers to ensure that the consumers is never overloaded with tasks and the incoming tasks are buffered in a queue. This improves the overall reliabiltiy of the system. In this article, we discussed some of the most popular messaging queues in use today and got an insight on their features.