I'm using Streaming Analytics to retrieve JSON messages being placed on an IoTHub. I need to be able to retrieve, and store, in a SQL Server table, the raw JSON string. I can't seem to find a way to do this. Is it possible?
No, unless you wrap the json as string in another json object that is send to the IoT Hub.
Otherwise the solution would be to try whether you can stringify / construct the json object using a Javascript function and then store it in the database.
Retrieve
Streaming Analytics Job does support retrieve JSON messages from IoT Hub. You can set the event serialization format to JSON type when adding stream input like this:
Store
Streaming Analytics Job support write to SQL Database.
But note:
Currently the Azure SQL Database offering is supported for a job
output in Stream Analytics. However, an Azure Virtual Machine running
SQL Server with a database attached is not supported. This is subject
to change in future releases.
So if you mean "SQL Server table" as Azure Virtual Machine running SQL Server with a database, the answer is "NO" currently.
Related
I am trying to get data from my Azure IoT hub to a database, which will be then displayed on a website.
I have seen many ways to ingest data such as azure data explorer cluster and stream analytics jobs. However, the issue i have with data explorer is that i cannot get connection details for the database so i cot connect to it using code. As for stream analytics jobs, MySQL databases are not supported, and I have only have experience with MySQL.
Does anyone know a way i can ingest data from my Azure IoT hub to a MySQL database? Thanks in advance.
You will have to write some code to do this.
You can for instance create an Azure Function that uses an IoTHubTrigger or EventHubTrigger so that the Function receives the messages that are available on the IoT Hub. In the Function, you write code to make sure that the data is inserted in the appropriate tables in your database.
Find some documentation regarding this here.
I am pushing events to my Event hub, then this data is being analyzed in Azure Stream Analytics. I'd like to visualize output from stream analytics in Grafana.
What is the easiest approach to achieve this?
Azure Stream Analytics job can natively ingest the data into Azure Data Explorer. https://techcommunity.microsoft.com/t5/azure-data-explorer/azure-data-explorer-is-now-supported-as-output-for-azure-stream/ba-p/2923654
You can then use the Azure Data Explorer plugin in Grafana. https://techcommunity.microsoft.com/t5/azure-data-explorer/azure-data-explorer-is-now-supported-as-output-for-azure-stream/ba-p/2923654
Another option is to use Power BI instead of Grafana. https://learn.microsoft.com/en-us/azure/stream-analytics/stream-analytics-power-bi-dashboard
If I remember correctly, Grafana doesn't store data locally, you need to define a data source on top of one of the compatible storage systems.
Azure Stream Analytics doesn't come with a storage layer either, it's compute only.
So if you want to use ASA and Grafana, you need to output data from ASA to a data source that is supported by Grafana in ingress.
Looking at both lists that leaves only MSSQL via Azure SQL (hopefully it's compatible) as a native option. It's not a bad option for narrow dashboards, or if you intend to store your data in a RDBMS anyway. You can store your entire payload in an NVARCHAR(MAX) if you don't plan to consume the data in SQL.
But being clever, we can actually use the Functions output to write to any other store, or call any API. I'm not sure if Grafana has a direct ingestion API, but Azure Monitor does and it's a supported data source in Grafana.
The other option would be to go through ADX as explained in the other answer.
Not straightforward but doable ;)
In my project I receive data from Azure IoThub and want to send it to a SQL database using Azure stream analytics. I'm trying to achieve this using the following query:
SELECT
IoTDataArrayElement.ArrayValue.sProjectID AS id
INTO
[test-machine]
FROM
[iothub-input] AS e
CROSS APPLY GetArrayElements(e.iotdata) AS IoTDataArrayElement
HAVING IoTDataArrayElement.ArrayValue IS NOT NULL
When I run the query in the environment provided by stream analytics and press test query I get the expected output which is a projectID. But when I start the stream analytics job the data doesn't go in to my database table. The table has 1 column 'id'.
When I try to send all the data to a blob storage the stream analytics job works.
Can someone please explain to me why the query I use for sending the data to a database doesn't actually send the data to a database?
Couple of things you need to verify to make successfully configuration of Azure SQL DB as output:
Make sure firewall settings is ON for All Azure Services.
Make sure you have configured the output to the sql database with the correct properties defined.
The following table lists the property names and their description for creating a SQL Database output.
Make sure the table schema must exactly match the fields and their
types in your job's output.
Hope this helps.
I am looking into combining data (stored in Azure SQL) and real-time stream data (coming via IoT Hub) in Stream Analytics. One way I found is to use blob storage to copy the SQL Azure data and use it as Input type "Reference Data" and in Stream Analytics query editor JOIN with the streaming data which works fine. However, I am looking into whether it is possible to use JavaScript UDF function capability in stream analytics to get data from SQL Azure and combine with streaming IoT data? I also don't know which one is the suggested approach to combining these type of data together?
Thanks
UDFs in streaming analytics won't allow you to call out to external services like SQL. They're used for things like basic data manipulation, regex, Math, etc. If your SQL data is slow moving in nature, the approach you've outlined here of using something like Data Factory to move SQL information into Blob storage and then use it as a Reference data inside your Stream Analytics query is the correct way (and only way currently) to solve your problem.
If it's fast moving data in SQL you'd want to investigate hooking into the SQL database changes and then publishing them on to Event Hubs. You could then pull this into your query as a second Data Stream input type and do the appropriate joins in your query.
I generated a SAS signature using this RedDog tool and successfully sent a message to Event Hub using the Events Hub API refs. I know it was successful because I got a 201 Created response from the endpoint.
This tiny success brought about a question that I have not been able to find an answer to:
I went to the azure portal and could not see the messages I created anywhere. Further reading revealed that I needed to create a storage account; I stumbled on some C# examples (EventProcessorHost) which requires the storage account creds etc.
Question is, are there any APIs I can use to persist the data? I do not want to use the C# tool.
Please correct me if my approach is wrong, but my aim is to be able to post telemetries to EventHub, persist the data and perform some analytics operations on it. The telemetry data should be viewable on Azure.
You don't have direct access to the transient storage used for EventHub messages, but you could write a consumer that reads from the EventHub continuously and persist the messages to Azure Table or to Azure Blob.
The closest thing you will find to a way to automatically persist messages (as with Amazon Kinesis Firehose vs Amazon Kinesis which EventHubs are basically equivalent to), would be to use Azure Streaming Analytics configured to write the output either to Azure Blob or to Azure Table. This example shows how to set up a Streaming Analytics job that passes the data through and stores it in SQL, but you can see the UI where you can choose a choice such as Azure Table. Or you can get an idea of the options from the output API.
Of course you should be aware of the requirements around serialization that led to this question
The Event Hub stores data for maximum of 7 days; that’s too in standard pricing tier. If you want to persist the data for longer in a storage account, you can use the Event Hub Capture feature. You don’t have to write a single line of code to achieve this. You can configure it through Portal or ARM template. This is described in this document - https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-capture-overview
The event hub stores it’s transient data in Azure storage. It doesn’t give any more detail in relation to the data storage. This is evident from this documentation - https://learn.microsoft.com/en-us/azure/event-hubs/configure-customer-managed-key
The storage account you need for EventProcessorHost is only used for checkpointing or maintaining the offset of the last read event in a partition.