Does anyone know if there is a library that allows you to send a message to a specific partition on an IoT hub with Azure.
I was previously able to achieve this with azuresblite library, unfortunatelly which I Cannot use.
https://github.com/ppatierno/azuresblite
There is no way to send a message to a specific partition. Period.
Partitions are used internally to allow scaling of the IoT (Event Hub) and allow for scaling out the consumer app (the one that reads the events out of the Hub).
While you can specify a Partition Key when using Event Hub, this does identify a Partition ID. And using IoT Hub, the option to specify Partition Key is anyway hidden and you cannot influence it. Specifying a Partition Key for a message in Event Hub will just make sure that all messages with the same partition key will fall into the same partition. But you cannot tell which partition (0,1,2..n). I have seen projects which try to literary abuse the Partitions and use them as "tenants". Very wrong way of completely abusing scalability of Event Hubs.
Please do not abuse the partitioning of Event Hub (which is anyway the back-end system for an IoT Hub). Do not force Partition Keys. Thus you are unbalancing the Event Hub and may go into a wrong direction. Instead please clearly define a technical issue you are trying to solve and we may be able to help you.
Azure IoT Hub ensures proper ordering of messages while messages are read from IoT Hub. PF the link for more details -
https://learn.microsoft.com/en-us/azure/iot-hub/iot-hub-devguide-messages-d2c#ordering-guarantees-with-at-least-once-delivery
Related
I have created an Azure function to route messages from an IoT hub to an Azure SQL DB using the IoTHubTrigger following mostly the instructions in this link Azure Functions - how to set up IoTHubTrigger for my IoTHub messages?.
Each IoT device captures data every 8 minutes. When capturing is done, the device streams the data in 4 different messages. Then the azure function takes over to write these 4 different messages to the database.
When only one device was streaming, I had no issues with the data, which where written in the db and I could also see/monitor events/messages using az iot hub monitor-events.
When a second device started streaming in the same IoT hub, I started missing messages, meaning that from each device only one message is being stored in the db. Also when using iot hub monitor-events only one message appears from each device. I was also expecting that if I disable the 2nd device, then the 1st one will go back to normal. Unfortynately the issue remains the same.
So my question is: how is it possible a 2nd device screwing up the way that the 1st one interacts with the hub?
If that's not the case, then how we are supposed to figure out what causes the problem at this stage?
Thanks :)
Difficult to say without more details. Are you routing messages in IoT Hub somewhere else? I would go back to a clean IoT Hub with one device and create a consumer group on the IoT Hub for the function. Before running the function I would monitor that consumer group (I like to use the Azure IoT Explorer application) to see if data is coming through as expected, then add another device and keep monitoring the same consumer group. If data is coming through then start the function (consuming data from the consumer group).
If telemetry was not getting read from the IoT Hub consumer group then you will need to look at your device code for any issues.
Right now, on IoT Hub there is an information that limit for messages per day 8000. I would like to ask you about any patterns which are being used in Azure.
I am curious if I am able to hit to Azure with some service outside Messages in order to prevent it from being overloaded by big amount of data, or save some confidentiality for this service.
For example, I would like to store some data from given service to Messages that are not being confidential and other data by using some WebSocket or any Rest protocol. I think that there are some patterns that serve that scenarios.
Does anyone has experience with that kind of situation?
Not everything needs to go through IoT Hub. IoT Hub is great for two way communication to/from IoT devices. You could also look at Event Hubs for ingestion from devices that don't need two way comms. We have a write up on the differences here Connecting IoT Devices to Azure: IoT Hub and Event Hubs.
I have a data pipeline set up in Azure where I send messages to an IoTHub which then routes those messages to an EventHub. When I read from the EventHub using the standard EventProcessorHost method, I find that only one of the partitions is being read from. I assume that only one partition is actually having messages routed to it. I have not specified a partition key anywhere and expect that the messages would be routed to all of the partitions of the event hub using round robin (as per the documentation at https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-programming-guide).
How can I configure my setup to route messages to all partitions of the event hub?
Like I said in the comment:
Is it possible you are only receiving data from one device? IoT Hub does automatic partitioning based on the deviceId, so the partition affinity might be the cause.
Our solution use Azure IoT hub to connect thousands of devices to our backend. Devices report operating state, and based on this state we control the devices. The challenge is that devices are installed in groups, and state changes from a device, affect other devices in the group. Because of this we need to handle messages for a group in sequence. I would have preferred to use the devices groupId to partition messages. IoT hub and its default endpoint use deviceId to partition messages, and we've had to find other means of synchronising messages across partitions. Up until now, we've used a semaphore for this, which have worked fine as we're running everything in one process.
As we're nearing the point where a single App Service plan can no longer handle all messages, we need to scale the solution out. Thus, the semaphore will no longer suffice, and we need to find an alternative to distribute and synchronise messages.
The plan is to use custom routing in IoT hub, forwarding messages to one or more event hub endpoints. Currently I see two options:
If it's possible to affect partition key when using custom routes/endpoints, we could assign each device a groupId, and use that for partitioning messages. This would route all messages for a group to the same processor without any additional synchronisation, allowing us to simply scale out the event processor to handle more messages. Sadly, I've not found a way to affect the partition key of messages when using custom routes/endpoints either, and it does not look like this is a viable solution.
Add multiple custom event hub endpoints to IoT hub, and use groupId to route messages to endpoints. This will require us to deploy multiple event processor instances, each configured to consume messages for "its" event hub. Since event hubs have a minimum of 2 partitions, we would still need to use a semaphore to synchronise messages destined for the event processor. This seems like the only viable option, but adds quite a bit of complexity to scaling as we would have to manually deploy and configure each processor instance, instead of simply scaling out the App Service plan and using partitions to distribute messages.
Are there ways to change partition key when using custom routes/endpoint, allowing us to implement solution 1., or are there other better ways to achieve this?
The simple way to split an IoT Hub stream into the sub-streams based on the sources (groupId) is to use an Azure function (AF), see the following screen snippet.
The AF sample:
[FunctionName("FunctionSplitter")]
public static async Task PreProcessor(
[EventHubTrigger("%IoTHubName%", Connection = "connectionIoTHub", ConsumerGroup = "splitter")] EventData ed,
[EventHub("%EventHubName%", Connection = "connectionEventHub")] IAsyncCollector<EventData> outputEventHubMessages,
TraceWriter log)
{
log.Info($"IoT Hub trigger function processed an event {ed.SequenceNumber}");
// Create a clone
var edclone = ed.Clone();
edclone.PartitionKey = Convert.ToString(ed.Properties.ContainsKey("deviceGroupId") ? ed.Properties["deviceGroupId"] : "1234567890");
await outputEventHubMessages.AddAsync(edclone);
await outputEventHubMessages.FlushAsync();
}
I have just started learning Azure IoT and it's quite interesting. I am confuse about does IoT Hub stores data somewhere?
i.e. Suppose i am passing room Temperature to IoT hub and want to store it in database for further use. How it's possible?
I am clear on how device-to-cloud and cloud-to-device works with IoT hub.
IoT Hub exposes device to cloud messages through an event hubs endpoint. Event Hubs has a retention time expressed in days. It's a stream of data that the reading client could re-read more time because the cursor is on client side (not on server side like queues and topics). With IoT Hub the related retention time is 1 day by default but you can change it.
If you want to store received messages from device you need to have a client reading on the Event Hubs exposed endpoint (for example with an Event Processor Host) that has the business logic to process the messages and store them into a database for example.
Of course you could use another decoupling layer so that the client reads from event hubs and store messages into queues. Then you have another client that at its own pace reads from queues and store into database. In this way you have a fast path reading event hubs.
This is pretty much the use case for all IoT scenarios.
Step 1: High scale data ingestion via Event Hub.
Step 2: Create and use a stream processing engine (Stream Analytics or HDInsight /Storm). You can run conditions (SQL like queries) to filter and store appropriate data in either cold or hot store for further analytics.
Step 3: Storage for cold-path analytics can be Azure BLOB. Stream Analytics can directly be configured to write the Data into it. Cold can contain all other data that doesn't require querying and will be cheap.
Step 4: Processing for hot-path analytics. This is data that is more regularly queries for. Or data where real time analytics needs to be carried on. Like in your case checking for Temperature values going beyond a threshold! needs an urgent trigger!
Let me know if you face any challenges while configuring the Stream analytics job! :)
If you take a look at the IoT Suite remote monitoring preconfigured solution (https://azure.microsoft.com/documentation/articles/iot-suite-remote-monitoring-sample-walkthrough/) you'll see that it persists telemetry in blob storage and maintains device status information in DocumentDb. This preconfigured solution gives you a working illustration of the points made in the previous answers.