My application (.NET-based) gets messages from a queue in a multithreaded fashion and I'm worried about the fact that I may receive messages in an out-of-order manner because one thread can be quicker than the other, for instance, given the following queue state:
[Message-5 | Message-4 | Message-3 | Message-2 | Message-1]
In a multithreaded operation, msg #2 may arrive before msg #1, even though msg #1 was first in the queue, due to many threading issues (thread time slices, thread scheduling etc).
In such a situation, it would be great if a message that is inside the queue have already stamped with an ordinal/sequence number when it was enqueued and even if I get the messages in an out of order fashion, I can still order them at some point within my application using their given ordinal-number attribute.
Any known mechanism to achieve it in a Websphere MQ environment?
You have 2 choices:
(1) Use Message Grouping in MQ as whitfiea mentioned or
(2) Change you application to be single threaded.
Note: If the sending application does not set the MQMD MsgId field then the queue manager will generate a unique number (based on queue manager name, date & time) and store it in the message's MQMD MsgID field.
You can obtain the MessageSequenceNumber from the MQMessage if the messages are put to the queue in a message group. The MessageSquenceNumber will either be the order that the messages were put to the queue by default or defined by the application that put the messages to the queue.
See the MessageSequenceNumber here for more details
Yes, if the originating message has an ordinal then as you receive your data you could:
Use a thread safe dictionary:
SortedDictionary<int,Message>
Related
I have created a queue in Azure Queue and enqueued two items in it. Using the nodejs sdk, i create a timer that executes every 5 secs and calls:
azure.createQueueService("precondevqueues", "<key>").getMessages(queueName, {numOfMessages : 1, visibilityTimeout: 1 }, callback)
I expect that the same message of the two in the queue to show up after every 5 secs but that does not seem to be the case. The output of this call alternates between the two messages.
This should not be the case since visibilityTimeout is set to 1 and hence, after 1 second, the message dequeued in the first call should be visible again before the next getMessage call is made.
As noted here, FIFO ordering is not guaranteed. So it may be the case, that most of the time messages are fetched in FIFO order, but that is not guaranteed and Azure can give you the messages in the order which is best for their implementation.
Messages are generally added to the end of the queue and retrieved
from the front of the queue, although first in, first out (FIFO)
behavior is not guaranteed.
Aha my mistake! I again read the getMessages documentation very carefully and realize that getMessages dequeues the message but retains a invisible copy outside of the queue. If the message processor does not delete the message before the visibility timeout expires, the copy is re-enqueued in the message and therefore they go to the end of the queue.
I need to limit the rate of consuming messages from rabbitmq queue.
I have found many suggestions, but most of them offer to use prefetch option. But this option doesn't do what I need. Even if I set prefetch to 1 the rate is about 6000 messages/sec. This is too many for consumer.
I need to limit for example about 70 to 200 messages per second. This means consuming one message every 5-14ms. No simultaneous messages.
I'm using Node.JS with amqp.node library.
Implementing a token bucket might help:
https://en.wikipedia.org/wiki/Token_bucket
You can write a producer that produces to the "token bucket queue" at a fixed rate with a TTL on the message (maybe expires after a second?) or just set a maximum queue size equal to your rate per second. Consumers that receive a "normal queue" message must also receive a "token bucket queue" message in order to process the message effectively rate limiting the application.
NodeJS + amqplib Example:
var queueName = 'my_token_bucket';
rabbitChannel.assertQueue(queueName, {durable: true, messageTtl: 1000, maxLength: bucket.ratePerSecond});
writeToken();
function writeToken() {
rabbitChannel.sendToQueue(queueName, new Buffer(new Date().toISOString()), {persistent: true});
setTimeout(writeToken, 1000 / bucket.ratePerSecond);
}
I've already found a solution.
I use module nanotimer from npm for calculation delays.
Then I calculate delay = 1 / [message_per_second] in nanoseconds.
Then I consume message with prefetch = 1
Then I calculate really delay as delay - [processing_message_time]
Then I make timeout = really delay before sending ack for the message
It works perfectly. Thanks to all
See 'Fair Dispatch' in RabbitMQ Documentation.
For example in a situation with two workers, when all odd messages are heavy and even messages are light, one worker will be constantly busy and the other one will do hardly any work. Well, RabbitMQ doesn't know anything about that and will still dispatch messages evenly.
This happens because RabbitMQ just dispatches a message when the message enters the queue. It doesn't look at the number of unacknowledged messages for a consumer. It just blindly dispatches every n-th message to the n-th consumer.
In order to defeat that we can use the prefetch method with the value of 1. This tells RabbitMQ not to give more than one message to a worker at a time. Or, in other words, don't dispatch a new message to a worker until it has processed and acknowledged the previous one. Instead, it will dispatch it to the next worker that is not still busy.
I don't think RabbitMQ can provide you this feature out of the box.
If you have only one consumer, then the whole thing is pretty easy, you just let it sleep between consuming messages.
If you have multiple consumers I would recommend you to use some "shared memory" to keep the rate. For example, you might have 10 consumers consuming messages. To keep 70-200 messages rate across all of them, you will make a call to Redis, to see if you are eligible to process message. If yes, then update Redis, to show other consumers that currently one message is in process.
If you have no control over consumer, then implement option 1 or 2 and publish message back to Rabbit. This way the original consumer will consume messages with the desired pace.
This is how I fixed mine with just settimeout
I set mine to process consume every 200mls which will consume 5 data in 1 seconds I did mine to do update if exist
channel.consume(transactionQueueName, async (data) => {
let dataNew = JSON.parse(data.content);
const processedTransaction = await seperateATransaction(dataNew);
// delay ack to avoid duplicate entry !important dont remove the settimeout
setTimeout(function(){
channel.ack(data);
},200);
});
Done
This is more of a conceptual question and doesn't apply to any particular programming language.
I have two entities communicating with each other, with three types of messages allowed:
Command Message: An unsolicited message commanding the other entity to do something.
Query Message: An unsolicited message asking the other entity for information.
Response Message: A solicited message answering a query message from the other entity.
Now each entity has two threads:
Reader Thread: Reads messages.
Worker Thread: Sends messages and does useful things
The two possible communication scenarios are:
Entity A sends a command to Entity B, and Entity A doesn't care what happens after.
Entity A sends a query to Entity B, and Entity A must wait until Entity B responds with the answer.
So the question is, how does the reader thread handle both solicited and unsolicited messages?
Unsolicited messages are easy to handle through events. The reader thread can just fire an event on the worker thread saying it received a query or a command, and the worker thread can react accordingly.
Solicited messages are hard to handle though. The worker thread sends a query, and must block until it receives a response or times out. How does the worker thread let the reader thread know it is waiting for a response, and how does the reader thread tie a response back to a specific query from the worker thread and deliver that response back to the worker thread's execution?
I know this has been done a million times in other programs, so whats the standard practice?
[I used Windows Azure Service Bus messaging entities as I am familiar with it, but in general this should be true with any Messaging system.]
Lets say your entity names are A and B.
Have 1 Topic (pub-sub entities) and 1 Queue for communication between A and B (as you need bidirectional communication) : Topic-A2B & Queue-B2A. A2B is for Commands from A to B or Queries from A to B and B2A, as the name says, is for Responses from B to A.
Typical Messaging Systems will offer MessageType property - for you to be able to set it and the later distinguish which type of messages you are reading and route it accordingly : Example from Windows Azure ServiceBus Brokered Message. Use that Property - to set whether its a Query or Command or Response.
The idea here is - while receiving a message in B - you will receive using Subscriptions. You will have 2 threads reading - (one) reads only Commands (theSecondOne) reads only Queries
For UnSolicited messages - as you said, its easy to handle. All you need to do is
A should send message to B with BrokeredMsg.ContentType="Cmd" and B should create a Subscription with a filter and read and process
For Solicited Messages - like Queries (a feature called Sessions will come handy here).
A should send Message to B with something like: BrokeredMessage.ContentType = "Query"
A also sets a correlation Id on the Message it sends to B: BrokeredMessage.SessionId = "ABC456" <-- The Correlation Id for A to be able to correlate this message with
Now A will wait for response and expects B to also set
BrokeredMessage.SessionId="ABC456" <--- The exact same value it had set earlier.
using the AcceptMessageSession API - with the Session Id and a Timeout. Ex: Q_B2A_QClient.AcceptMessageSession("ABC456", 2 mins)
At the receiving end B should Create a Subscription with a filter to be able to Receive these messages.
Once B receives the query - it processes and puts back the result in the Q-BToA
If B succeeds to put back the message in the Q-B2A in less than 2 Mins - then A will receive it and then you can orchestrate it further with a Callback method (as all of these are async methods - you will not need to use any Reader or Writer thread as you mentioned above - which will be a huge performance booster).
HTH!
Sree
It's known that, in cases when one needs comunicate between UI thread and working thread, an hidden window must be created because of thread safety(handle reconstruction).
For exemplify:
Form1 has N dynamicaly created TProgressBar instances with the same name of a background running .
Is always garanteed that WM_REFRESH will only be called inside Task Thread.
Form1 has H : THandle property that allocates the following procedure:
procedure RefreshStat(var Message: TMessage); message WM_REFRESH;
Inside RefreshStat, in cases when there is only 1 background thread I could easily use L and W parameter to map Task Id and position.
I don't know if the title says what I want to know, but let's imagine if we have an application that has multiple background tasks running.
In my case I use TProgressBar to report progress the done.
Does AllocateHwnd garantee that all messages arrives with no race condition the hidden window?
What happens if two or more tasks post the message at the same time?
If this needs to be controled manually, I wonder if there is something else to do besides creating another message loop system in the custom message.
I hope the question is clear enough.
The message queue associated with a thread is a threadsafe queue. Both synchronous and asynchronous messages from multiple other thread are delivered safely no harmful date races. There is no need for any external synchronization when calling the Windows message API functions like SendMessage and PostMessage.
If two threads post or send messages to the same window at the same time, then there is no guarantee as to which message will be processed first. This is what is known as a benign race condition. If you want one message to be processed before the other then you must impose an ordering.
I'm considering a multi-threaded architecture for a processing pipeline. My main processing module has an input queue, from which it receives data packets. It then performs transformations on these packets (decryption, etc.) and places them into an output queue.
The threading comes in where many input packets can have their contents transformed independently from one another.
However, the punchline is that the output queue must have the same ordering as the input queue (i.e., the first pulled off the input queue must be the first pushed onto the output queue, regardless of whether its transformations finished first.)
Naturally, there will be some kind of synchronisation at the output queue, so my question is: what would be the best way of ensuring that this ordering is maintained?
Have a single thread read the input queue, post a placeholder on the output queue, and then hand the item over to a worker thread to process. When the data is ready the worker thread updates the placeholder. When the thread that needs the value from the output queue reads the placeholder it can then block until the associated data is ready.
Because only a single thread reads the input queue, and this thread immediately puts the placeholder on the output queue, the order in the output queue is the same as that in the input. The worker threads can be numerous, and can do the transformations in any order.
On platforms that support futures, they are ideal as the placeholder. On other systems you can use an event, monitor or condition variable.
With the following assumptions
there should be one input queue, one output queue and one working queue
there should be only one input queue
listener
output message should contain a wait
handle and a pointer to worker/output data
there may be an arbitrary number of
worker threads
I would consider the following flow:
Input queue listener does these steps:
extracts input message;
creates output message:
initializes worker data struct
resets the wait handle
enqueues the pointer to the output message into the working queue
enqueues the pointer to the output message into the output queue
Worker thread does the following:
waits on a working queue to
extract a pointer to an output
message from it
processes the message based on the given data and sets the event when done
consumer does the following:
waits on n output queue to
extract a pointer to an output
message from it
waits on a handle until the output data is ready
does something with the data
That's going to be implementation-specific. One general solution is to number the input items and preserve the numbering so you can later sort the output items. This could be done once the output queue is filled, or it could be done as part of filling it. In other words, you could insert them into their proper position and only allow the queue to be read when the next available item is sequential.
edit
I'm going to sketch out a basic scheme, trying to keep it simple by using the appropriate primitives:
Instead of queueing a Packet into the input queue, we create a future value around it and enqueue that into both the input and output queues. In C#, you could write it like this:
var future = new Lazy<Packet>(delegate() { return Process(packet); }, LazyThreadSafetyMode.ExecutionAndPublication);
A thread from the pool of workers dequeues a future from the input queue and executes future.Value, which causes the delegate to run JIT and returns once the delegate is done processing the packet.
One or more consumers dequeues a future from the output queue. Whenever they need the value of the packet, they call future.Value, which returns immediately if a worker thread has already called the delegate.
Simple, but works.
If you are using a windowed-approach (known number of elements), use an array for the output queue. For example if it is media streaming and you discard packages which haven't been processed quickly enough.
Otherwise, use a priority queue (special kind of heap, often implemented based on a fixed size array) for the output items.
You need to add a sequence number or any datum on which you can sort the items to each data packet. A priority queue is a tree like structure which ensures the sequence of items on insert/pop.