I would appreciate your thoughts on this.
Node app 1 sends data to a RabbitMQ queue. The data contains a unique ID.
Node app 2 requests data with a specific ID from the RabbitMQ queue.
So as you can see, I need to be able to select specific messages from the queue, rather than just the next available message.
Is this possible? How can I do it?
Thanks.
Yes. You can use either header or topic exchange - look for Exchanges and Exchange Types here. For topic there is also tutorial here.
no directly from a single queue
if you have 3 messages in a queue, those messages will come out of that queue in order: first in, first out.
the "selective consumer" pattern for retrieving a message by some value, from a queue, is an anti-pattern in rabbitmq.
to accomplish what you want, you need to create an exchange / queue / binding setup that sends your message to a specific queue so that your specific consumer can handle it.
Related
I'm creating a consumer of an Azure Service Bus topic (subscription) that does nothing but store some statistics. The messages sent to the topic contains a rather large body, that is handled by another consumer on a second subscription on the same topic.
Since the statistics consumer can handle a large number of messages in one go, I was wondering if it is possible to receive a lot of messages but leave out the body to improve performance when communicating with Service Bus and to receive even more messages in one go.
I'm currently doing this:
this.messageReceiver = new MessageReceiver(conn, path);
...
await messageReceiver.ReceiveAsync(10, TimeSpan.FromSeconds(5));
It works pretty sweet but it would be nice to be able to receive 100 or more messages, without having to worry about moving large messages over the network.
Before anyone suggests it, I already know that I can ask for a count, etc. on a topic subscription. I still need the Message object since that contains an entry in the UserProperties dictionary that is used to calculate the stats.
Not possible. You can peek, but that brings the whole payload and headers w/o incrementing the DeliveryCount of the message. You could request it as a broker feature here.
I would like to know if I can have persistence in my Spring Integration setup when I use a aggregator, which is not backed by a MessageStore, by leveraging the persistence of AMQP (RabbitMQ) queues before and after the aggregator.
I imagine that this would use ack's: The aggregator won't ack a message before it's collected all the parts and sent out the resulting message.
Additionally I would like to know if this is ever a good idea :)
I am new working with queue's, and am trying to get a good feel for patterns to use.
My business logic for this is as follows:
I receive a messages on one queue.
Each message must result in two unrelated webservice calls (preferably in parallel).
The results of these two calls must be combined with details from the original message.
The combination must then be sent out as a new message on a queue.
Messages are important, so they must not be lost.
I was/am hoping to use only one 'persistent' system, namely RabbitMQ, and not having to add a database as well.
I've tried to keep the question specific, but any other suggestions on how to approach this are greatly appreciated :)
What you would like to do recalls me Scatter-Gather EI Pattern.
So, you get a message from the AMQP send it into the ScatterGather endpoint and wait for the aggregated reply. That's enough for to stick with the default acknowledge.
Right, the scatterChannel can be PublishSubscribeChannel with an executor to call Web Services in parallel. Anyway the gatherer process will wait for replies according the release strategy and will block the original AMQP listener do not ack the message prematurely.
Quite new to RabbitMQ and I'm trying to see if I can achieve what I need with it.
I am looking for the Worker Queues pattern but with one caveat. I want to have only a single worker running concurrently per routing key.
An example for clarification:
If i send the following messages with routing keys by order: a, a, b, c, I want to have only 3 workers running concurrently. When the first a message is received a worker picks it up and handles it.
When the next a message is received and the previous a message is still handled (not acknowledged) the new a message should wait in queue. When the b and c messages are received they each get a worker handling them. When the first a message is acknowledged any worker can pick up the next a message.
Would that pattern be possible using RabbitMQ in a natural way (without writing any application code on my side to handle the locking and stuff...)
Edit:
Another clarification. All workers can and should handle all messages, and I don't want to have a queue per Worker as I want to share the load between them, and the Publisher doesn't know which Worker should process the message. But I do want to make sure that no 2 Workers are working on messages sharing the same key at the same time.
For example, if I have a Publisher publishing messages with a userId field, I want to make sure no 2 Workers are handling messages with the same userId at the same time.
Edit 2
Expanding on the userId example. Let's say I have a single Publisher and 3 Workers. The publisher publishes messages like these: { userId: 1, text: 'Hello' }, with varying userIds. My 3 Workers all do the same thing to this messages, so I can have any of them handle the messages coming in. But what I'm trying to achieve is to have only a single worker processing a message from a certain user at the same time. If a Worker has received a message with userId 1 and is still processing it, and another message with userId 1 is received I want to make sure no other Worker picks up that message. But other messages coming in with different userIds should be processed by other available Workers.
userIds are not known beforehand, and the publisher doesn't know how many workers are or anything specific about them, he just wants to schedule the messages for processing.
what your asking is not possible with routing keys, but is built into queues with a few settings.
if you define "queue_a" for a messages, "queue_b" for b messages, etc, you can then have as many consumers connect to it as you want.
RabbitMQ will only deliver a given message to a single consumer of a given queue.
The way it works with multiple consumers on a single queue is basic round-robin style dispatch of the messages. that is, the first message will be delivered to one of the consumers, and the next message (assuming the first consumer is still busy) will be delivered to the next consumer.
So, that should satisfy the need to deliver the message to any given consumer of the queue.
To ensure your messages have an equal chance of getting to any of the consumer (and are not all delivered to the same consumer all the time), there are a few other settings you should put in place.
First, make sure to set the message consumer no ack setting to false (sometimes called "auto ack"). This will force you to ack the message from your code.
Lastly, set the "consumer prefetch" limit of the consumer to 1.
With this combination of settings, a single consumer will retrieve a single message and begin working on it. While that consumer is working, any message waiting in the queue will be delivered to other consumers if any are available. If there are none available, the message will wait in the queue until a consumer is available.
With this, you should be able to achieve the behavior you are wanting, on a given queue.
...
Keep in mind this only applies to queues, though. routing keys cannot be managed this way. all matched routing keys from an exchange will cause a copy of the message to be sent to the destination queue.
I am using Node.js with node-amqp to create a simple message queue. Most examples that I see do the following:
Create a connection
Create an exchange
Creat a Queue and Bind it to the Exchange
Publish via the Exchange
In my code, I omit the queue (step 3) since it is not used for publishing.
var _connection = amqp.createConnection(_options);
_connection.on('ready', function() {
_connection.exchange('myexchange', { type: 'direct', autoDelete: false }, function(ex) {
ex.publish({hello:'world'});
});
});
Is this ok? or is there a reason for the queue?
There is nothing wrong with the code that you have. This is a good example of how you can keep your message producer nice and simple / clean.
However, the code you've shown is only half of the messaging solution. You need both a message producer, as shown, and a message consumer.
A Message Consumer
The message consumer is the code that does the real work. It receives a message from a queue to which it is subscribed, and processes that message however you tell it to.
That's the key, here - a message consumer will consume a message from a queue. If you want to send a message and have it be processed, then, you must have a queue for the message.
The Postal System Analogy
Think of this like this:
When you write a letter (pen and paper), you put it in an envelope. Then you write an address on the envelope and send it through your postal system. The postal system knows what the address means, sends it through various trucks and mail processing centers, and eventually puts it in a mailbox for the recipient.
It's the same thing with messaging in RabbitMQ.
You are delivering a letter to a destination. You write an "address" (exchange name, and routing key) on the message and RabbitMQ figures out how to deliver it to the appropriate place.
With physical mail, your letter is put in a mailbox for someone to read. With RabbitMQ and messaging, your message it put in a queue for some software to read.
You need a queue for the software to receive the message and process it.
...
P.S. If you're in need of some ground-up materials on RabbitMQ and NodeJS, check out my RabbitMQ For Developers package. It will get you up and running in no time, with the most common RMQ questions and patterns.
Queues are explicitly created and bound to exchange to make sure published message will not be lost in case no queue(s) and bindings previously exists.
In RabbitMQ most operations about entities creations (exchanges, queues, bindings) are idempotent. It means that if you call them more than once with the same arguments, they will provide the same result as called once.
In case of exchange, you can't publish to nonexistent exchange (channel-level AMQP exception occurs), but if no proper queues and bindings exist for particular message, it will be lost (or dead-lettered, see Dead Letter Exchanges and Alternate Exchanges for more).
We are thinking of speparate Queues for:
Request (RequestQueue)
Response (ResponseQueue)
Scenario:
Worker role will putMessage to RequestQueue e.g. GetOrders
Third party will monitor RequestQueue. If they see GetOrders
request they will getMessage, process them and put the response in
ResponseQueue.
Question:
If I putMessage to RequestQueue, I will like to get results back from ResponseQueue. Is there easy way to achieve this and how?
Thank you.
No, this is not possible. If you put a message in a queue, you must pop the message from the same queue (it will not magically appear in any other queue). Perhaps if you explained more why you think you need two separate queues here for push/pop, there might be a more expansive answer and suggestion.
EDIT: Perhaps I misunderstood your intent. I guess I don't get the question now - can you help clarify. You seem to be asking how to put a message on one queue, acknowledge it by putting another message on another queue, and have someone read the acknowledgment from the second queue? What is the question here? I should point out that you won't want some 3rd party to read directly from a Windows Azure queue as that would require sharing the master storage key with them (a non-starter). Perhaps you are looking for how to have 3rd parties read from a queue?
EDIT 2: Sounds like you want to consume messages with a 3rd Party. Windows Azure queues probably are not a good fit as I mentioned due to security reasons (you need to share the master key). Instead, you could either layer a WCF service over the queue (using queues via proxy) or use the queueing from the Service Bus - that will allow you to have separate credentials. Using the Service Bus capability might be the right choice here in terms of simplicity. Take a look here for demos.
Have a worker of some sort monitor the question queue, then post an answer to the answer queue. Interface out the queue managers and you shouldn't have any problems using any sort of queue tech. Also, the worker doesn't really need to use a queue for answers..
Caveats:
Worker service has access to both queues
Each queue item contains a serialized foreign key to identify themselves.