I have a spring-integration application, which uses a message-driven-channel-adapter to receive a xml message from a Websphere MQ queue and then passing this message to a spring integration channel (comprised ) for storing into database.
How do I make sure only 1 message is processed at a time, that is no message can be processed if the preceding message has not reached a specific endpoint (in my case, a service-activator) ?
If you don't use any queue or executor channels, and you leave the default adapter concurrency to 1, the entire flow will run on the single container thread.
Hence, only one message will be processed at a time.
Related
I am new to message queues. I read the documentation of rabbitmq today, about the setting of "prefetch_count", this is to set the number of unpacked messages that consumers can have the most. I looked at the official documentation and said that the order is guaranteed even if re-enqueued. Assuming a single consumer, the producer sends the message 1->2->3->4->5->6->7->8->9->10, if the prefetch_count is set to 30, then if the 5 message fails and it does't has manual ack, this failed message is requeue. However, 6, 7, 8, 9, 10 are successfully consumed and all acked. At this time, although the messages seem to be pushed to consumers in order, the order of consuming messages maybe 1->2->3-> 4->6->7->8->9->10->5 order. I have this question:
In the scenario where the order of consumption needs to be guaranteed, how do you all do the above situations in the real scenario?
If the order of consumption needs to be guaranteed, how to ensure that the next message will not be consumed if the previous message is not successfully consumed?
Is it necessary to persist the message to MySql? For example, record the message of consumption failure
Here is the rabbitmq doc about the order.
Message ordering guarantees
Section 4.7 of the AMQP 0-9-1 core specification explains the conditions under which order is guaranteed: messages published in one channel, passing through one exchange and one queue, and one outgoing channel will be received in the same order that they were sent. RabbitMQ offers stronger guarantees since release 2.7.0.
Messages can be returned to the queue using AMQP methods that feature a requeue parameter (basic. recover, basic. reject, and basic. nack), or due to a channel closing while holding unacknowledged messages. Any of these scenarios caused messages to be queued at the back of the queue for RabbitMQ releases earlier than 2.7.0. From RabbitMQ release 2.7.0, messages are always held in the queue in publication order, even in the presence of queueing or channel closure.
With release 2.7.0 and later it is still possible for individual consumers to observe messages out of order if the queue has multiple subscribers. This is due to the actions of other subscribers who may require messages. From the perspective of the queue the messages are always held in the publication order.
I have a peculiar type of problem statement to be solved.
Configured RabbitMQ as message broker and its working but when there is failure in process in consume I'm now acknowledging with nack but it blindly re-queues with whatever already came in as payload but i want to add some-more fields to it and re-queue with simpler steps
For Example:
When consume gets payload data from RabbitMQ it will then process it and try to do some process based on it in multiple host machines, but due to some thing if one machine not reachable i need to process that alone after some time .
Hence I'm planning to re-queue failed data with one more fields with machine name again back to queue so it will be processed again with existing logic itself.
How to achieve this ? Can someone help on me
When a message is requeued, the message will be placed to its original position in its queue, if possible. If not (due to concurrent deliveries and acknowledgements from other consumers when multiple consumers share a queue), the message will be requeued to a position closer to queue head. This way you will end up in an infinite loop(consuming and requeuing the message). To avoid this, you can, positively acknowledge the message and publish it to the queue with the updated fields. Publishing the message puts it at the end of the queue, hence you will be able to process it after some time.
Reference https://www.rabbitmq.com/nack.html
We are on JBoss EAP 6.4.
There is a JMS queue that has a consumer who immediately consumes everything that is placed to the queue.
I want to check what is placed to the queue but for my tool (Hermes JMS) the queue is always empty.
I tried configuring fine level logging for "org.hornetq" category but messages placed to the queue are not logged at any debug level.
Is there a way to see what is placed to the queue when it is immediately consumed?
Thanks,
Valery
If you don't need to see the content but only see the flow, you could execute read-resource command, or create a custom script that reads the queue and certain times and calculate the flow. You would just need to parse it and calculate, doing your own tool, simple and efficient.
CLI command to read jms queue
/subsystem=messaging/hornetq-server=default/jms-queue=testQueue/:read-resource(recursive=false,proxies=false,include-runtime=true,include-defaults=true)
Reference for CLI
Quite new to RabbitMQ and I'm trying to see if I can achieve what I need with it.
I am looking for the Worker Queues pattern but with one caveat. I want to have only a single worker running concurrently per routing key.
An example for clarification:
If i send the following messages with routing keys by order: a, a, b, c, I want to have only 3 workers running concurrently. When the first a message is received a worker picks it up and handles it.
When the next a message is received and the previous a message is still handled (not acknowledged) the new a message should wait in queue. When the b and c messages are received they each get a worker handling them. When the first a message is acknowledged any worker can pick up the next a message.
Would that pattern be possible using RabbitMQ in a natural way (without writing any application code on my side to handle the locking and stuff...)
Edit:
Another clarification. All workers can and should handle all messages, and I don't want to have a queue per Worker as I want to share the load between them, and the Publisher doesn't know which Worker should process the message. But I do want to make sure that no 2 Workers are working on messages sharing the same key at the same time.
For example, if I have a Publisher publishing messages with a userId field, I want to make sure no 2 Workers are handling messages with the same userId at the same time.
Edit 2
Expanding on the userId example. Let's say I have a single Publisher and 3 Workers. The publisher publishes messages like these: { userId: 1, text: 'Hello' }, with varying userIds. My 3 Workers all do the same thing to this messages, so I can have any of them handle the messages coming in. But what I'm trying to achieve is to have only a single worker processing a message from a certain user at the same time. If a Worker has received a message with userId 1 and is still processing it, and another message with userId 1 is received I want to make sure no other Worker picks up that message. But other messages coming in with different userIds should be processed by other available Workers.
userIds are not known beforehand, and the publisher doesn't know how many workers are or anything specific about them, he just wants to schedule the messages for processing.
what your asking is not possible with routing keys, but is built into queues with a few settings.
if you define "queue_a" for a messages, "queue_b" for b messages, etc, you can then have as many consumers connect to it as you want.
RabbitMQ will only deliver a given message to a single consumer of a given queue.
The way it works with multiple consumers on a single queue is basic round-robin style dispatch of the messages. that is, the first message will be delivered to one of the consumers, and the next message (assuming the first consumer is still busy) will be delivered to the next consumer.
So, that should satisfy the need to deliver the message to any given consumer of the queue.
To ensure your messages have an equal chance of getting to any of the consumer (and are not all delivered to the same consumer all the time), there are a few other settings you should put in place.
First, make sure to set the message consumer no ack setting to false (sometimes called "auto ack"). This will force you to ack the message from your code.
Lastly, set the "consumer prefetch" limit of the consumer to 1.
With this combination of settings, a single consumer will retrieve a single message and begin working on it. While that consumer is working, any message waiting in the queue will be delivered to other consumers if any are available. If there are none available, the message will wait in the queue until a consumer is available.
With this, you should be able to achieve the behavior you are wanting, on a given queue.
...
Keep in mind this only applies to queues, though. routing keys cannot be managed this way. all matched routing keys from an exchange will cause a copy of the message to be sent to the destination queue.
Question: How to roll back messages in a Message Store back to a queue if a system failure occurs?
Scenerio:
IBM MQ Queue --> JMS Message Drive Channel Adapter--> Aggregator.
During a system failure that results in the a JVM shut down. Instead of using the default volatile message store could I specify a MQ queue to act as the persistent message store? That way if a system failure occurs the messages in the message store could be rolled back a MQ persistent queue for fail over handling.
EDIT: Last sentence for clarification.
It's not clear what you mean by...
could be rolled back to the originating queue
...if you mean automatically then, no.
There's a JDBC message store implementation (but you'd need XA transactions).
You could probably create a custom message store backed by JMS, but it's an unusual use of a queue - I suppose you'd have to use a message selector based on the groupId but you'd probably have to keep a mirror in memory (and initialize it during system initialization, while leaving the messages in the queue).