I have my IoT data flow as below:
Azure IoT Hub(JSON Data)->Stream Analytics->Azure CosmosDB Table API(Table name= Test)
The input to the Streaming Analytics is from IoT Hub(sensor data in JSON format). And I have two outputs from the Streaming Analytics, one is Azure CosmosDB(Table API, Table name is 'Test1' and another output is configured to Blob Storage. The streaming analytics query used for both outputs is same.
Q. I am able to see the data in azure ablob storage container, but I am not able to see the data in 'Test1' table of Azure CosmosDB(Table API) through portal. Please find attached screen shots for reference.
Thanks,
Siva M
Azure CosmosDB TableAPI
ASA Output
Right now Azure Streaming analytics does not support outputting to Azure Cosmos DB Table API.
Thanks,
Yaron
Related
I am looking for way to automatically export data from a Log Analytics table into an Azure SQL database table? Does anyone know how to do this on an automated schedule?
To export log analytics data to Blob storage or ADLS, you can try one of below.
Log Analytics – data export (preview) and example
Archive data from Log Analytics workspace to Azure storage using Logic App
Next ....You can go for Ingest option in ADF.
Next, configure source linked service i.e. Blob Storage or Azure Table storage linked and Sink i.e. SQL DB
Or any relevant source or sink option you want as per the logs stored.
Req: This actually for export last 6 months data from Timeseries insights to CSV file to download to User from web application.
The historical data is available on Blob & Timeseries insights.
Approach:
Create azure data factory pipeline which pulls the data from azure time series insights based on aggregate/filter query and convert it into CSV to store in Blob storage.
Is it possible to do that or any other approach can be considered for this requirement.
Please suggest.
Thanks
Data Factory doesn't support Azure Time Series Insights as the dataset(Source or Sink).
Ref the connector overview.
That means we can't create azure data factory pipeline which pulls the data from azure time series insights based on aggregate/filter query and convert it into CSV to store in Blob storage.
Some other way, you may could using Azure Blob storage client library to create and upload the file to Blob Storage.
Here's the tutorials may be helpful:
Query data from the Azure Time Series Insights Gen2 environment
using C Sharp
Quickstart: Azure Blob storage client library v12 for .NET
Getting error messages. Seems this is not possible ?
message is :
The JSON provided in the request body is invalid. Property 'server' value 'postgres-mydatabase123.postgres.database.azure.com' is not acceptable.
According to the documentation this is not possible as currently the support output sinks are
Azure Data Lake Store
SQL Database Blob storage
Event Hub
Power BI
Table Storage
Service Bus Queues
Service Bus Topics
Azure Cosmos DB
Azure Functions (In Preview)
Out of all of these Azure Functions or Event Hub might be interesting to you as they allow custom code to process the data. In your case that would be sending it to the PostgreSQL database.
Currently as part of our requirements we are working with the below Azure components
Azure Event Hub
Azure Stream Analytics
Azure Table Storage
Azure Sql DB
Basically with first 3 components, we will be building an Analytics and Reports platform.
Currently as we just started we analyze the data from Azure Table Storage and display it in the analytics dashboard.
Recently we came across a new Azure product Azure Data Lake . Doing some research on microsoft website , we could see we can easily migrate data from Azure Table Storage (with help of Azure Data Factory) to Azure Lake Store. Creating big data pipelines using Azure Data Lake and Azure Data Factory
As we go through the above link, it's mentioned that we need to create an Azure Data Lake Analytics pipeline to process the data.
So what am unclear is the where will be analytics output data will be saved. Do we need to save the analytics output to some DB ? or can we real-time analytics through a Http request ?
We have huge number rows of records in Azure Table Storage that will be moved to Azure Data Lake. For this scenario is it a good option or Can we go an analytics-based solution from Azure Table Storage itself.
Please share your thoughts
You can store your analytics output data on Azure Data Lake Store (a data repository that enables you to store all kinds of data in their raw format without defining schemas.) after processing it through Azure Data lake Analytics (An analytics service that enables you to run jobs on data sets without having to think about clusters.)
As you said "We have huge number rows of records in Azure Table Storage that will be moved to Azure Data Lake.", I think performing analytics on data placed on Azure data lake store is much more efficient because it offers unlimited storage with immediate read/write access to it and scaling the throughput you need for your workloads. It's also offers small writes at low latency for big data sets. So I believe it is better choice then Azure Table storage.
What is the best way to send data from Event Hubs to Data Lake Store?
I am assuming you want to ingest data from EventHubs to Data Lake Store on a regular basis. Like Nava said, you can use Azure Stream Analytics to get data from EventHub into Azure Storage Blobs. Thereafter you can use Azure Data Factory (ADF) to copy data on a scheduled basis from Blobs to Azure Data Lake Store. More details on using ADF are available here: https://azure.microsoft.com/en-us/documentation/articles/data-factory-azure-datalake-connector/. Hope this helps.
==
March 17, 2016 update.
Support for Azure Data Lake Store as an output for Azure Stream Analytics is now available. https://blogs.msdn.microsoft.com/streamanalytics/2016/03/14/integration-with-azure-data-lake-store/ . This will be the best option for your scenario.
Sachin Sheth
Program Manager, Azure Data Lake
In addition to Nava's reply: you can query data in a Windows Azure Blob Storage container with ADLA/U-SQL as well. Or you can use the Blob Store to ADL Storage copy service (see https://azure.microsoft.com/en-us/documentation/articles/data-lake-store-copy-data-azure-storage-blob/).
One way would be to write a process to read messages from the event hub event hub API and writes them into a Data Lake Store. Data Lake SDK.
Another alternative would be to use Steam Analytics to get data from Event Hub into a Blob, and Azure Automation to run a powershell that would read the data from the blob and write into a data lake store.
Not taking credit for this, but sharing with the community:
It is also possible to archive the Events (look into properties\archive), this leaves an Avro blob.
Then using the AvroExtractor you can convert the records into Json as described in Anthony's blob:
http://anthonychu.ca/post/event-hubs-archive-azure-data-lake-analytics-usql/
One of the ways would be to connect your EventHub to Data Lake using EventHub capture functionality (Data Lake and Blob Storage is currently supported). Event Hub would write to Data Lake every N mins interval or once data size threshold is reached. It is used to optimize storage "write" operations as they are expensive on a high scale.
The data is stored in Avro format, so if you want to query it using USQL you'd have to use an Extractor class. Uri gave a good reference to it https://anthonychu.ca/post/event-hubs-archive-azure-data-lake-analytics-usql/.