I try to receive messages from Devices with the "EventHubReceiver" (Device2Cloud). Each device should have it's own, single receiver.
It isn't a problem to create one single EventHubReceiver (per Partition) for all devices:
string iotHubconnectionString = CloudConfigurationManager.GetSetting("Microsoft.IotHub.ConnectionStringPayed");
string iotHubD2cEndpoint = "messages/events";
EventHubClient eventHubClient = EventHubClient.CreateFromConnectionString(iotHubconnectionString, iotHubD2cEndpoint);
EventHubRuntimeInformation runtimeInformation = eventHubClient.GetRuntimeInformation();
If I then want to receive messages from the clients I do the following steps:
EventHubReceiver eventHubReceiver = eventHubClient2.GetDefaultConsumerGroup().CreateReceiver(partition); //Get the partitions via the runtimeInformation: string[] partitions = runtimeInformation.PartitionIds;
var incommingMessage = eventHubReceiver.ReceiveAsync(); //Wait here for incomming messages
That works all right, but all messages from all "Devices" arrive then at this one "EventHubReceiver". I would like to have multiple receiver, that only receive messages from a single device.
I tried to change the following codeline:
string iotHubD2cEndpoint = "messages/events";
to
string iotHubD2cEndpoint = "devices/{deviceID}/messages/events";
but that doesn't work all right. I get the following error:
The link address 'devices/secondDevice/messages/events/$management' did not match any of the expected formats. Supported formats: '/$cbs', '/devices/{deviceid}/messages/events', '/devices/{deviceid}/messages/deviceBound', '/messages/deviceBound', '/messages/serviceBound/feedback', '/messages/events/*'.
So the problem is that I get 'devices/secondDevice/messages/events/$management' insted of 'devices/secondDevice/messages/events/'
I don't know, whether it is just not possible to create a single EventHubReceiver for each Device or I've got a error in the code or thinking.
When a device send telemetry data to IoT Hub, the events are made available to the related D2C endpoint in the cloud that is "event hub compatible".
It has an "event hub like" behaviour and for this reason we are able to get message using an EventHubReceiver.
However, event hubs works in partitions and the incoming message can be assigned to partitions in a round robin fashion or hashing a partition key.
In the IoT Hub architecture I don't know if it use round robin but it's possibile that it use to hash the device id (as partition key) so all messages from a device go into a partition. It doesn't mean that the partition contains messages only for that device ! It's impossible to have a partition for each device :-)
So ... a partition containes messages mixed from different device (but message from a specific device go always in the same partition). An Event Hub receiver can read from partition so it gets all messages from more device. You need to distinguish them based on device id.
I forgot to mention, that I've already got a solution. But it is not very nice:
All deviceThreads (one for each partition and device) wait ("WaitOne") at one point (locked with AutoResetEvent)
I receive all messages with a single receiver
I put the message into a own queue for each device (Dictionary that contains as key a object of the device(id and partition) and as value a List
I set a single thread set free with the "set" command.
The thread looks in its queue and:
if there is a message in the queue it continues and returns the result with a "yield return" -> waits again for a new message
else it sets a different thread fee and returns to the freeze state
All threads loop in a while(true) to wait for messages.
The solution is working, but it doesn't look very performant (with many threads) and a bit complicated.
Related
When publishing messages to a service bus topic, if I loop over 3 messages:
{ A, B, C }
And await the SendAsync() each time, I'd expect them to be published to the topic in the order:
{ A, B, C }
public async Task PublishMessage(string topic, string json, string sessionId)
{
var topicClient = new TopicClient(_connectionString, topic);
var busMessage = new Message(Encoding.UTF8.GetBytes(json));
busMessage.SessionId = sessionId;
await topicClient.SendAsync(busMessage);
}
A number of employees have suggested this isn't guaranteed to be the case, and that in certain scenarios (i.e large messages), this publishing order isn't guaranteed. I've never encountered a scenario of this myself, does ASB not guarantee publish ordering even when the sending of messages is awaited like the above?
This article https://devblogs.microsoft.com/premier-developer/ordering-messages-in-azure-service-bus/ uses this quote:
"While Azure Service Bus allows for a FIFO approach (First-In-First-Out), we cannot guarantee that messages are entered in the order we want them to be processed"
This all seems quite baffling to me, as I'd have assumed SendAsync() would only return a successful result once the message has been added into the topic. Do we really need to write layers of complexity around this to manage it?
Please note this only relates to the publishing of messages, we use SessionIds to handle consumption.
Even if you wait for one message to be sent and then only send the next one, the FIFO is not guaranteed. This is due to too many probable causes. In order to ensure you get guaranteed ordering, you need to use session enabled queues or subscriptions.
I try to get 32 messages per request from Azure Queue.
queue.ApproximateMessageCount;
This code gives me the result of 1509. Telling me the connection is OK and it has records. Also I check in queue it really has 1509 records.
But when I try to retrieve records I don't get any record.
I do the following:
var messages = await queue.GetMessagesAsync(configuration.MessageBatchSize);
if (!messages.Any()) {
return;
}
It always goes in the if and returns.
What is going on here and what am I missing?
Do do that, receiving messages in batch mode, i use this kind of code :
var messages = await queueClient?.ReceiveBatchAsync(Max_Messages);
foreach (var message in messages)
{
await dispatcher.Dispatch(message); // do something with each message
}
But, for receiving messages with ReceiveBatchAsync, the queue have to be configured with the EnableBatchedOperations flag to true.
ApproximateMessageCount property represents the total number of messages available in queue at that particular moment. It does not represent that all messages (max #32 messages in a pull) are ready to be dequeued. You can use this property to infer that how many messages are in queue.
queue.ApproximateMessageCount;
If you could not retrieve the message by, GetMessagesAsync(numberOfMessages), then it says that all messages are not available or invisible for current QueueClient.
var cloudQueueMessages = await cloudQueue.GetMessagesAsync(numberOfMessages);
You could try polling the queue after sometime to see if messages came back to surface.
Note that, be advised of setting adequate visibility timeout for any message being dequeued to avoid indefinite starvation :)
This is more of a conceptual question and doesn't apply to any particular programming language.
I have two entities communicating with each other, with three types of messages allowed:
Command Message: An unsolicited message commanding the other entity to do something.
Query Message: An unsolicited message asking the other entity for information.
Response Message: A solicited message answering a query message from the other entity.
Now each entity has two threads:
Reader Thread: Reads messages.
Worker Thread: Sends messages and does useful things
The two possible communication scenarios are:
Entity A sends a command to Entity B, and Entity A doesn't care what happens after.
Entity A sends a query to Entity B, and Entity A must wait until Entity B responds with the answer.
So the question is, how does the reader thread handle both solicited and unsolicited messages?
Unsolicited messages are easy to handle through events. The reader thread can just fire an event on the worker thread saying it received a query or a command, and the worker thread can react accordingly.
Solicited messages are hard to handle though. The worker thread sends a query, and must block until it receives a response or times out. How does the worker thread let the reader thread know it is waiting for a response, and how does the reader thread tie a response back to a specific query from the worker thread and deliver that response back to the worker thread's execution?
I know this has been done a million times in other programs, so whats the standard practice?
[I used Windows Azure Service Bus messaging entities as I am familiar with it, but in general this should be true with any Messaging system.]
Lets say your entity names are A and B.
Have 1 Topic (pub-sub entities) and 1 Queue for communication between A and B (as you need bidirectional communication) : Topic-A2B & Queue-B2A. A2B is for Commands from A to B or Queries from A to B and B2A, as the name says, is for Responses from B to A.
Typical Messaging Systems will offer MessageType property - for you to be able to set it and the later distinguish which type of messages you are reading and route it accordingly : Example from Windows Azure ServiceBus Brokered Message. Use that Property - to set whether its a Query or Command or Response.
The idea here is - while receiving a message in B - you will receive using Subscriptions. You will have 2 threads reading - (one) reads only Commands (theSecondOne) reads only Queries
For UnSolicited messages - as you said, its easy to handle. All you need to do is
A should send message to B with BrokeredMsg.ContentType="Cmd" and B should create a Subscription with a filter and read and process
For Solicited Messages - like Queries (a feature called Sessions will come handy here).
A should send Message to B with something like: BrokeredMessage.ContentType = "Query"
A also sets a correlation Id on the Message it sends to B: BrokeredMessage.SessionId = "ABC456" <-- The Correlation Id for A to be able to correlate this message with
Now A will wait for response and expects B to also set
BrokeredMessage.SessionId="ABC456" <--- The exact same value it had set earlier.
using the AcceptMessageSession API - with the Session Id and a Timeout. Ex: Q_B2A_QClient.AcceptMessageSession("ABC456", 2 mins)
At the receiving end B should Create a Subscription with a filter to be able to Receive these messages.
Once B receives the query - it processes and puts back the result in the Q-BToA
If B succeeds to put back the message in the Q-B2A in less than 2 Mins - then A will receive it and then you can orchestrate it further with a Callback method (as all of these are async methods - you will not need to use any Reader or Writer thread as you mentioned above - which will be a huge performance booster).
HTH!
Sree
My application (.NET-based) gets messages from a queue in a multithreaded fashion and I'm worried about the fact that I may receive messages in an out-of-order manner because one thread can be quicker than the other, for instance, given the following queue state:
[Message-5 | Message-4 | Message-3 | Message-2 | Message-1]
In a multithreaded operation, msg #2 may arrive before msg #1, even though msg #1 was first in the queue, due to many threading issues (thread time slices, thread scheduling etc).
In such a situation, it would be great if a message that is inside the queue have already stamped with an ordinal/sequence number when it was enqueued and even if I get the messages in an out of order fashion, I can still order them at some point within my application using their given ordinal-number attribute.
Any known mechanism to achieve it in a Websphere MQ environment?
You have 2 choices:
(1) Use Message Grouping in MQ as whitfiea mentioned or
(2) Change you application to be single threaded.
Note: If the sending application does not set the MQMD MsgId field then the queue manager will generate a unique number (based on queue manager name, date & time) and store it in the message's MQMD MsgID field.
You can obtain the MessageSequenceNumber from the MQMessage if the messages are put to the queue in a message group. The MessageSquenceNumber will either be the order that the messages were put to the queue by default or defined by the application that put the messages to the queue.
See the MessageSequenceNumber here for more details
Yes, if the originating message has an ordinal then as you receive your data you could:
Use a thread safe dictionary:
SortedDictionary<int,Message>
I'm beginning a project which will involve Azure Queue (not Service Bus).
I'm trying to figure out what is the best practice to find out whether there are messages waiting in the Queue.
AFAIK, there are two methods for that:
Using the ApproximateMessageCount property of the Queue object
Calling GetMessage, and if the returned value is null - there are no messages.
Which one is better performance-wise? Is there any difference?
From a billing POV, I understand there is a transaction cost for both of them, is that correct?
Thanks!
GetMessage is both faster and cheaper. GetMessage is also more correct from a logic perspective since the message count will return both messages that have already been retrieved by another reader as well as messages that have expired without being deleted.
I have also used this code in the past:
var cnnString = "the connection string";
var queueName = "the queue name";
var nsManager = NamespaceManager.CreateFromConnectionString(cnnString);
return nsManager.GetQueue(queueName).MessageCount;
That said - this was from about 4 months ago.
Any reason you need to do this (i.e. are you not just consuming messages off the queue?)