I created a Event Hub with 32 partitions and doing the message injection over the IoT Hub message routing.
If i connect with Stream Analytics to my Event Hub and look at the input i get this message: "While sampling data, no data was received from '31' partitions"
I thought that if i have multiple partitions all the messages will be distributed over all the partitions and not land in one.
If this is intended, then whats the use to have multiple partitions when using Stream Analytics with Event Hub as input?
IoT Hub will use the device ID as the partition key I think.
So if you receive data only from one device, they will all go to one partition of the Event Hub.
You'll get more usage of the partitions by having more devices sending data to IoT Hub.
Related
I've got data coming into IoTHub and want to filter on them.
Relevant data I want to forward to slack as notification.
I've got the IoT Hub and a slack subscription in place and am having trouble connecting the two.
In order to do a rather complex time-based query, I figure to use Stream Analytics and configure the IoT Hub as input. From research I found Logic Apps can send messages to Slack over a webhook. Using a Service Bus Queue as output for Stream Analytics, I can get the data into Logic Apps.
So it's:
IoT Hub (ingest all data) => Stream Analytics (filter) => Service Bus Queue (queue up the data) => Logic Apps (send to Slack)
Looks a bit bulky but that seems to be one way of doing it (is there a better one?!?)
Doing this I ran into issues. I selected my IoT Hub as input for Stream Analytics and the simple query SELECT * INTO [Queue] FROM [Hub] fails, saying there was no data.
It does make sense if the IoT Hub just pushes new data to its endpoints and then discards it. So I created a test set in the Stream Analytics Job and the query runs fine.
However I do get data into the Hub which is not (all) picked up nor forwarded by the job to the service bus queue. I do see some activity on the queue but not nearly enough to be the data I receive.
This seems to be a very common scenario, ingesting data in IoT Hub and sending notifications to email or slack if they are of a certain type. Can you explain the steps to take or point me to a resource that does it. Maybe I'm on the wrong path as I cannot find anything that describes this.
Thanks
I have an Azure IoT device connected to an Azure IoT Hub. The device sends 6 - 7 messages per minute. By looking at the D2C message metrics, I found an outlier, that states that at a specific time, the count of the D2C message was zero (see picture). As the messages are routed to a storage, I can check the storage to see if there are messages missing at that specific time, but the data saved in the storage shows that every message was received correctly at that time. Does any one know how that comes or if the metrics are not that reliable generally? If that's the case, what is the best practice to monitor the IoT Hub message transfer?
IoT Hub D2C Message Metrics
EnqueuedTimeUtc in the storage
For precisely monitor the flow of each message through IoT Hub you will need to Trace Azure IoT device-to-cloud messages with distributed tracing (currently in preview)
This trace context includes correlation IDs that allow you to correlate events from one component with events from another component
Automatically log the trace context to Azure Monitor Logs.
Measure and understand message flow and latency from devices to IoT Hub and routing endpoints.
Ref: https://learn.microsoft.com/en-us/azure/iot-hub/iot-hub-distributed-tracing
I have a device which is sending data via IOT hub to Eventhub and then to other data layers. The data is in JSON format. I could identify some delay to reach the data to my Database, compared to the time the packet is actually generated at the device(We have a time in the JSON packet). I need to identify the ingestion time of a packet when it reaches IOT hub to analyze and find where the delay is happening. How we can do that?
Every message gets a couple of System Properties added by IoT Hub. What you are looking for is iothub-enqueuedtime
I have a data pipeline set up in Azure where I send messages to an IoTHub which then routes those messages to an EventHub. When I read from the EventHub using the standard EventProcessorHost method, I find that only one of the partitions is being read from. I assume that only one partition is actually having messages routed to it. I have not specified a partition key anywhere and expect that the messages would be routed to all of the partitions of the event hub using round robin (as per the documentation at https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-programming-guide).
How can I configure my setup to route messages to all partitions of the event hub?
Like I said in the comment:
Is it possible you are only receiving data from one device? IoT Hub does automatic partitioning based on the deviceId, so the partition affinity might be the cause.
When creating a new Azure IOT Hub you are asked how many device-to-cloud partitions you need. You can select between 2-32 partitions for standard tiers.
I understand that the SKU and number of units determine the maximum daily quota of messages that you can send to IOT Hub. And that it is recommended to shard your devices into multiple IOT hubs to smooth traffic bursts. However, device-to-cloud partitions need clarification.
1>> What is the purpose of those device-to-cloud partitions under a single IOT hub?
2>> How are we supposed to take advantage of those IOT Hub device-to-cloud partitions?
Thanks.
1>> What is the purpose of those device-to-cloud partitions under a
single IOT hub?
Partition property is setting for Event Hub-compatible messaging endpoint(messages/events) built in Azure IoT Hub. From here we can see "partitions" is a concept belongs to Event Hub.
Event Hubs is designed to allow a single partition reader per consumer group. A single partition within a consumer group cannot have more than 5 concurrent readers connected at any time. More partitions enables you to have more concurrent readers processing your data, improving your aggregate throughput.
Ref: Built-in endpoint: messages/events and How many partitions do I need?
2>> How are we supposed to take advantage of those IOT Hub
device-to-cloud partitions?
Event Hubs has two primary models for event consumption: direct receivers and higher-level abstractions, such as EventProcessorHost. Direct receivers are responsible for their own coordination of access to partitions within a consumer group.
Ref:Event consumers.
More information about the partitioning model of Azure Event Hubs are here.