I've got data coming into IoTHub and want to filter on them.
Relevant data I want to forward to slack as notification.
I've got the IoT Hub and a slack subscription in place and am having trouble connecting the two.
In order to do a rather complex time-based query, I figure to use Stream Analytics and configure the IoT Hub as input. From research I found Logic Apps can send messages to Slack over a webhook. Using a Service Bus Queue as output for Stream Analytics, I can get the data into Logic Apps.
So it's:
IoT Hub (ingest all data) => Stream Analytics (filter) => Service Bus Queue (queue up the data) => Logic Apps (send to Slack)
Looks a bit bulky but that seems to be one way of doing it (is there a better one?!?)
Doing this I ran into issues. I selected my IoT Hub as input for Stream Analytics and the simple query SELECT * INTO [Queue] FROM [Hub] fails, saying there was no data.
It does make sense if the IoT Hub just pushes new data to its endpoints and then discards it. So I created a test set in the Stream Analytics Job and the query runs fine.
However I do get data into the Hub which is not (all) picked up nor forwarded by the job to the service bus queue. I do see some activity on the queue but not nearly enough to be the data I receive.
This seems to be a very common scenario, ingesting data in IoT Hub and sending notifications to email or slack if they are of a certain type. Can you explain the steps to take or point me to a resource that does it. Maybe I'm on the wrong path as I cannot find anything that describes this.
Thanks
Related
I don't quite understand how Microsoft is calculating my Evenhub costs.
The current setup is:
I have a Raspberry PI Zero which sends messages once a minute via https to an Azure IOT Hub and the IOT Hub routes it to the Event Hub. One Message seems to be around 2kb big as seen in the Event Hub Throughput Graph. The Eventhub is then read by Elastic Logstash which uploads the messages into Elasticsearch. So we have 2kb/min incoming and outgoing traffic.
A raw message looks like:
{ "humidity":98.86653465006785,
"#timestamp":"2021-02-12T01:07:05.883Z",
"pressure":1035.0542695256731,
"#version":"1",
"temperature":-10.694375312741613
}
This is only 149 Bytes in total. I got the number through putting it into a txt file and taking a look into the properties.
My Service is now running since three days and already consumed 0,68$ which seems too much in my opinion.
If I interpret the MS Azure Event Hub pricing page corectly it charges me with 0,015$/h for generating traffic at 1MB/s incoming and 2mb/s outgoing
Did I make a mistake or is there a way of lowering the costs?
For anyone looking up to this question, there is indeed a way to reduce the cost of operation. In my case I used the IOT Hub to redirect the messages to the Event Hub, which is nonsense in my case. The Event Hub is totally unneccesary for this. You can use the IOT Hub like an Event Hub to get your messages.
I have created an Azure function to route messages from an IoT hub to an Azure SQL DB using the IoTHubTrigger following mostly the instructions in this link Azure Functions - how to set up IoTHubTrigger for my IoTHub messages?.
Each IoT device captures data every 8 minutes. When capturing is done, the device streams the data in 4 different messages. Then the azure function takes over to write these 4 different messages to the database.
When only one device was streaming, I had no issues with the data, which where written in the db and I could also see/monitor events/messages using az iot hub monitor-events.
When a second device started streaming in the same IoT hub, I started missing messages, meaning that from each device only one message is being stored in the db. Also when using iot hub monitor-events only one message appears from each device. I was also expecting that if I disable the 2nd device, then the 1st one will go back to normal. Unfortynately the issue remains the same.
So my question is: how is it possible a 2nd device screwing up the way that the 1st one interacts with the hub?
If that's not the case, then how we are supposed to figure out what causes the problem at this stage?
Thanks :)
Difficult to say without more details. Are you routing messages in IoT Hub somewhere else? I would go back to a clean IoT Hub with one device and create a consumer group on the IoT Hub for the function. Before running the function I would monitor that consumer group (I like to use the Azure IoT Explorer application) to see if data is coming through as expected, then add another device and keep monitoring the same consumer group. If data is coming through then start the function (consuming data from the consumer group).
If telemetry was not getting read from the IoT Hub consumer group then you will need to look at your device code for any issues.
in our usecase, we receive messages on Azure IoT Hub, and would like to route the data to different Event Hubs or Service Bus topics.
IoT Hub routes and endpoints are no option, because the data is binary data (protobuf), and there are only 10 different endpoints possible (we need more).
Our requierements are:
Splitting the message
Transform the data (maybe json)
Routing to different endpoints based on the payload (different parts of message could be routed to different endpoints)
(optional) enrich the data with additional payload
I see different options:
Azure Stream Analytics
Azure Functions
Spark or Flink
Do it yourself (write an Application and run it in Service Fabric or Kubernets)
Which techology would you recommend?
Regards,
Markus
There is also another option for your scenario such as using an Azure Event Grid. In this case, the telemetry data from the Azure IoT Hub are pushed to the Event Grid via its custom topic endpoint. Note, that there is a limit for the event message such as 64KB, see more details here.
The Event Grid allows to subscribe unlimited number of the Event Hubs, more details about the Event Grid are here and here.
Based on the above, the following screen snippet shows your another option for routing a small telemetry data to more than 10 Event Hubs, basically to any kind of subscriber.
I would like to get some recommendation, for designing a routing of IoT messages in Azure.
I have following scenario:
Senors sending messages to Azure IoT Hub in Google Protobuf format. Depending of the type of a message, I want to route the message to different applications inside a service fabric.
My current approach is to use a service fabric application to receive all messages from the IoT hub, parse the protobuf message, send the message depending on their type (attribute inside the protobuf) to an type-specific Azure event hub. Now the applications fetches the messages from their "own" event hub and process the messages.
I'm not sure if this is the best approach. I don't like the fact, to have one event hub for each type of message. Service Bus Topics are probably not an option, because I have a lot of messages (~30k per second).
Do I realy need a event hub, to decoupling this process, or does it make sense, to send the messages from the "routing application" direct to the different "type applications"?
What do you think?
Regards,
Markus
If you really need high performance you should take a look at IoT Hub and Event Hubs. Azure Event Hubs is a highly scalable data streaming platform and event ingestion service capable of receiving and processing millions of events per second. Event Hubs can process and store events, data, or telemetry produced by distributed software and devices. Data sent to an event hub can be transformed and stored using any real-time analytics provider or batching/storage adapters.
In other hand if you need only 30k messages per second you can go with Premium Messaging.
Comparison of Azure IoT Hub and Azure Event Hubs
Premium Messaging: How fast is it?
What is Event Hubs?
I am currently working on Internet Of Things, in my current project I was implemented the logic for to send temperature values to IoT Hub (using Raspberry PI2 and BMP280 sensor), in the azure part I created stream analytics job for receiving the messages from IoT Hub and filters those values based on my query like if temperature value exceeds 30 deg and post those filtered values to event hub is one of the output of the stream analytics job.
Query I wrote in stream analytics job.
SELECT
System.timestamp AS Time,
DeviceId,
RoomTemp,
RoomPressure,
RoomAlt
INTO
eventhub
FROM
bmpsensordata
WHERE RoomTemp>35
I was already created one event hub in azure, and monitor those filtered values by stream analytics job, in dashboard of event hub what I was created earlier.
But I want to send the values (filtered values by stream analytics job) of event hub to IoT Hub from that I will receive the values in the form of alert message/ notification using UWP App(C# language).
Please tell me how I can do it above scenario.
Regards,
Pradeep
I think that your solution could look like in the diagram I prepared:
Once data is retrieved from the IoT Hub and analyzed by Stream Analytics it can call Azure Function which triggers Azure Notification Hub to send push notification to your UWP application.
Please use my tutorial to see how to use Stream Analytics together with Azure function and at the end how to send SMS alerts - in your case you should replace the code with the one for Notification Hub:
https://github.com/Daniel-Krzyczkowski/Daniel-Krzyczkowski.github.io/blob/master/cloudyofthings/article1/index.md
Here is the documentation how to use Notification Hub SDK and how to integrate it with UWP applications:
https://learn.microsoft.com/en-us/azure/notification-hubs/notification-hubs-aspnet-backend-windows-dotnet-wns-notification
I think you will need another Stream Analytics job with Event Hub as input, and IoT Hub as output. Then you can receive the cloud-to-device messages from IoT Hub in your UWP application as described in this article.
You can have multiple output from a single Stream Analytics job. Refer to https://blogs.msdn.microsoft.com/streamanalytics/2015/09/16/query-pattern-of-the-week-send-data-to-multiple-outputs/ for more information on this.
Stream Analytics does not have a direct output to IoT hub though.
You'd need to put the info in to an EventHub and have a worker role process this and send the info from there in to IoT hub
Per my experience, I think you can try to integrate Notification Hub with IoTHub, Stream Analytics, Event Hub to implement your needs. Please see the details below.
Create a Stream Analytics Job with IoTHub as input and Event Hub as output for filtering sensor data.
Create a Notification Hub for pushing data to UWP app.
Create an server service or a scheduler job for receiving and sending the data from the Event Hub to the Notification Hub, such as continuous WebJob.
As references, there are some documents which show you how to do it.
Get started with Azure Stream Analytics to process data from IoT devices
, https://azure.microsoft.com/en-us/documentation/articles/stream-analytics-get-started-with-azure-stream-analytics-to-process-data-from-iot-devices/
Getting started with Notification Hubs for Windows Store Apps, https://azure.microsoft.com/en-us/documentation/articles/notification-hubs-windows-store-dotnet-get-started/
Event Hubs programming guide, https://azure.microsoft.com/en-us/documentation/articles/event-hubs-programming-guide/
Notification Hub Server SDK reference for .NET, https://msdn.microsoft.com/library/mt414893.aspx
Create a .NET WebJob in Azure App Service (run continuously), https://azure.microsoft.com/en-us/documentation/articles/websites-dotnet-webjobs-sdk-get-started/
Hope it helps.
Any concern, please feel free to let me know.