I am having some issues actually retrieving and using the data I send to the IoT Hub in Azure. When I run 'az IoT hub monitor-events --hub-name ' in CLI I can see my events, and I can also send messages to my devices in the IoT hub.
I have then tried to create a stream, to forward the messages to my SQL database, but without any luck. Do you have any suggestions on how to retrieve this data?
There are multiple ways about this. The two most common scenarios are probably using an Azure Function, or using a Stream Analytics job. I don't know what you've tried up until this point, but a Stream Analytics job is probably the easiest way to go.
Stream Analytics
This answer on SO could be what you're looking for, it also links to this tutorial that you could follow from "Create a new Azure SQL Database" onwards. It covers creating an IoT Hub input and Azure SQL output on your Stream Analytics job and using a simple query to link the two together. There is more info in the Microsoft docs here
Azure Function
While looking this one up I found this answer, which is mine, awkward. But it describes how you can go about creating an Azure Function that accepts IoT Hub messages and shoots them to your database. This option is a lot more cost-efficient (or even free, if you use the consumption plan for a Function) for a few devices.
Related
So, I am working on an IoT solution on Azure, we have been using a partner solution where we had the partner's devices linked to his cloud solution that exposes the data to us Via REST services. Right now we want to have our own IoT Cloud Solution on Azure.
At first, I am planning to build a Bridge between our IoT Solution and the partner's cloud solution via its REST Services that will link to our IoT Hub in order to ingest the data to our cloud.
Also, the data will not be only telemetry data but we'll have to send commands as well to those devices.
My question: I would like to know what would be the appropriate technology/solution to use a gateway (Data Grid, Azure Function, Azure WebJob)
The numbers in the picture represent the step that I am considering to tackle this problem.
1- First we are implementing an Application gateway that will have to get the data from the partner's system and sending commands to their system. It will allow us to first build the other components of our system and make sure that it can handle what is in place right now.
2- Second, the partner's devices will connect directly to a device gateway that is connected to our IoT Hub. In this case, we will not be using the gateway made in 1 anymore.
3- Finally, we will have our own devices connected to our IoT Hub, the partner's devices will always be connected to our IoT Hub via the gateway built in 2.
Let me try to answer your questions in the order you have asked.
For application gateway, where you are trying to pull data through
REST, you can use Azure functions and then you use Cosmos DB or any
storage to save data. I see , after getting device data from Partner
network, you are routing it to IoT-Hub (I would not say, its
incorrect), however once we pull data through Rest, we can directly
put into DB. So my Answer is to use Azure functions to pull data
from Partner solutions and put into DB.
If partner device is capable of running Azure IoT sdks or can be
provisioned to send data to IoT Hub directly, this will ease lot of
things and you would be able to send D2C and C2D messages easily.
further, here you can route data to DB by using configuration from
IoT Hub.
For your devices you can use IoT Hub Directly or can use Azure
IoT Edge (device gateway as you pointed ), both are fine , depends
on use case and also if we want to do some edge computation or
analytics at device side. And one important suggestions, use Azure
functions where ever you find that you have to integrate devices
data through Rest. Most cost effective in such scenarios.
Let me know if it clears your doubts.
After some time working on the subject, I did implement an AZURE Function app for the following reasons :
Supports Continuous Deployment and Integration Even though Azure Functions is serverless architecture, it still supports Continuous Deployment and Continuous Integration
Capabilities for implementing code - Being event-driven, the application platform has capabilities to implement code triggered by events occurring in any third-party service or on-premise system.
Compute-on-demand: This delivery model ensures that computing resources are available to the users as per their demand.
I have also used Azure Table Storage as database storage technology.
We are developing an application where IoT devices will be publishing events to azure IoT hub using MQTT protocol (by using one topic to push message). We want to consume these message using Stream Analytic service. And to scale Stream analytic services, it is recommended to use partitionBy clause.
Since, we are not using Azure Event hub SDK, can we somehow attached partitionId with events?
Thanks In Advance
As Rita mentioned in the comments, Event Hub will automatically associate each device to a particular partition.
Then, when you can use PARTITION BY PartitionId for steps closer to the input to efficiently parallelize processing of the input and reduce/aggregate the data.
Then, you can have another non-partitioned step to output to SQL sending some aggregate data.
Doing that you will be able to assign more thank 6 SUs, even with an output to SQL.
We will update our documentation to give more info about scaling ASA jobs and describe the different possible scenarios.
Thanks,
JS - Azure Stream Analytics
I'd like to build a small solution on Azure for practice. I'd be sending data using IOT-HUB from some devices and what I need is some way to interpret this data and do appropriate query to Azure SQL.
Basically I would need a way to have my program running all the time being able to:
listen to events from iot-hub
interpret event information and save/get data to/from database
send a message to some device using iot-hub
Which service would be good for that? Am I able to use Entity Framework?
In ideal solution I'd create a C# program to do what I need and have it running in Azure, waiting for events from iot-hub, having access to my database - is it even possible?
I'm very new (rather completely new) to cloud solutions, so I'd be really grateful for any advices. Currently I feel completely lost in all these Azure services.
There is quite some documentation on Azure IoT and how one can possible architect an IoT solution.
The IoT documentation is the obvious first step to get an overview of what Azure offers. There are some nice 'Getting Started' walkthroughs also
Take a look at this IoT reference architecture. Quite helpful to get an overview.
There are tons of links and interesting examples for Azure IoT. Just google around.
Im trying to create a complete solution to present data from IoT devices on to a webpage.
The data and devices will never be in the millions so using Stream Analytics, Machine Learning, Big Data etc. is costly and unnecessary.
I've looked at docs, blogs, forums for weeks now, and im stuck with the part on how to process the messages that the IoT hub receives, i want to save them to a SQL database and then build a website that will present them to the users.
What i have so far:
1. Device part
Raspberry Pi 3 has Windows IoT Core installed
Messages are sent and recieved on both Hub and Device ends successfully
(verified with Device Explorer and IoT hub dashboard)
2. Processing part
The most similar approach is detailed here but i don't want to use NoSQL, ive tried to use the Azure Function with the External Table (experimental) but there is zero documentation for that and all my attempts failed with function error.
Now im trying to connect a WebJob to process IoT Hub messages but i cant find any relevant samples or docs. Essentially id want to convert a Console App to a WebJob which will be triggered when a message arrives to the IoT hub
3. Webpage part
Once i get the messages to the SQL database i will create my custom portal for managing and registering devices, issuing one-off commands to devices and for request-response data.
The telemetry will be queried from the database and presented statically or near real time (with SignalR) by device type, location, by user privilages etc. this part is preety clear to me.
Please can anyone help me out with the processing part??
I found a solution by using Azure WebJobs and this article explains how to tie an EvenHub (IoT Hub) to the WebJob.
Does Steam Analytics support input sources other than products in the Azure family?
For example, can I setup a REST endpoint and send events this way? Are there client libraries for node.js?
Documentation is somewhat scant in this regard; I wanted to check here before assuming no on both fronts.
I believe the answer is no Azure Stream Analytics does not currently support non Azure sources.
One recommended approach is to write to Azure Event Hub then let Azure Stream Analytics read from there.
You could write to an event hub in Node.JS:
http://hypernephelist.com/2014/09/16/sending-data-to-azure-event-hubs-from-nodejs.html
Revise for my old answer.
As #PanagiotisKanavos said, Azure Stream Analytics (ASA) is just the processing service engine, not the ingestion endpoint, that doesn't need to have a non-azure input source as EventHub do and that how to feed ASA with data to it.
EventHub can be used by ASA, has a variety of libraries that work on tons of different machines, form factors etc, and can run on any OS and many frameworks. Worst case, simple HTTP works as well, AMQP is not mandatory but definitely ideal in terms of performance.
The correct route is PRODUCER -> EventHub -> ASA or PRODUCER -> STORAGE -> ASA. So if there is a library that supports storage on the device that they want, it can work as well, but EventHub is obviously a better choice.
Thanks a lot for #PanagiotisKanavos help.
Some circumstantial evidence below seems to prove that Azure not suppport non Azure Service as input for Stream Analytics.
From the REST API Create Input of Stream Analytics https://msdn.microsoft.com/en-us/library/azure/dn835010.aspx, there are only three data sources that include Event Hub, Blob Storgae & IoT Hub.
Screenshots from Azure old & new portal for add input.
Fig 1. The input options on Azure old portal (Step 1)
Fig 2. The options for Data stream (Step 2)
Fig 3. The option for Reference data (Step 2)
Fig 4. The input options on Azure new portal