In the context of Azure IoT hub, when would one use Stream Analytics over Time Series Insights?
The product pages and documentation for both indicates they are heavily geared for IoT/data applications. However, I'm not clear on the differences.
The use case I have is both real time monitoring as well as ETL analysis. Could (or even should?) the two be used together?
One immediate difference I can see is that Time Series Insights stores the data whereas Stream Analytics (I think) would need the developer to integrate storage.
In short, stream analytics is about transforming, filtering and aggregation of data and time series insight is about visualising (stored) data.
Data passed through stream analytics is typically forwarded to resources like power bi (for realtime monitoring) or storage like a database for later analysis or processing.
One immediate difference I can see is that Time Series Insights stores the data whereas Stream Analytics (I think) would need the developer to integrate storage.
This is a correct statement. TSI is a data store, but its purpose is to create an environment to (visually) analyze that data. ASA cannot be used to analyze data on its own.
You could use ASA to transform the data and have the data send to Event Hub. That same Event Hub can then be used as a data source for TSI.
Related
I have a device which sends data in the form of Hexadecimal value based on an algorithm to IoT Hub. I want to parse that data into a json string to store it in Cosmos DB. Is there any way to achieve this?
I would like to add to Roman's comment on your question, he's right to offer this link to Stream Analytics. It will get the job done. Depending on how many devices you have and how often you are receiving telemetry, you might want to consider using Azure Functions instead. See this sample on how to integrate Azure Functions between IoT Hub and CosmosDB.
The reason I offer this extra solution is that a Stream Analytics Job will cost you a fixed price per hour per streaming unit, while a Function is paid by consumption. Because the conversion from hexadecimal is a fairly small Function, you might even use it for free whereas a Stream Analytics Job in West Europe will cost at least 74 Euros.
We have legacy applications that currently write out various run time metrics (SQL calls run time, api / http request run times etc) to local SQL DB.
format:( source, event, data, executionduration)
We are moving away from storing those in local SQL DB, and are now publishing those same metrics to azure event hub.
Looking for a good place to store those metrics for the purpose of monitoring the health of the application. Simple solution would be to store in some DB and build custom application to visualize the data in custom ways.
We are also considering using Azure Monitor for this purpose via data collector API (https://learn.microsoft.com/en-us/azure/azure-monitor/platform/data-collector-api)
QUESTION: Are there any issues with azure monitor that would prevent us from achieving this type of health monitoring?
Details
each event is small (few hundred characters)
expecting ~ 10 million events per day
retention of 1-2 days is enough
ability to aggregate old events per source per event is important (to have historical run time information)
Thank you
You can do some simple graphs and with the Log Analytics query language, you can do just about any form of data analytics you need.
Here's a pretty good article on Monitor Visualizations.
learn.microsoft.com/en-us/azure/azure-monitor/log-query/charts
I noticed that both Azure Functions and Azure Stream Analytics can take an input, modify or transform that input, and put it into an output.
When would I use one versus the other? Are there any general rules I can use to decide?
I tried looking at the pricing of each to guide me, but I'm having trouble discerning how my logic would affect the compute time cost of Functions versus the App service plan cost of Functions versus the streaming unit cost of Stream Analytics.
Azure Stream Analytics is a real time analytics service which can "run massively parallel real-time analytics on multiple IoT or non-IoT streams of data" whereas Azure Function is a (serverless) service to host functions (little pieces of code) that can be used for e.g. event-driven applications.
General rule is always difficult since everything depends on your requirement, but I would say if you have to analyze a data stream, you should take a look at Azure Stream Analytics and if you want to implement something like a serverless event-driven or timer-based application, you should check Azure Function or Logic Apps.
Working on a IoT telemetry project that receives humidity and weather pollution data from different sites on the field. I will then apply Machine Learning on the collected data. I'm using Event Hubs and Stream Analytics. Is there a way of pulling the data to Azure Machine Learning without the hassle of writing an application to get it from Stream Analytics and push to AML web service?
Stream Analytics has a functionality called the “Functions”. You can call any web service you’ve published using AML from within Stream Analytics and apply it within your Stream Analytics query. Check this link for a tutorial.
Example workflow in your case would be like the following;
Telemetry arrives and reaches Stream Analytics
Streaming Analytics (SA) calls the Machine Learning function to apply it on the data
SA redirects it to the output accordingly, here you can use the PowerBI to create a predictions dashboards.
Another way would be using R, and here’s a good tutorial showing that https://blogs.technet.microsoft.com/machinelearning/2015/12/10/azure-ml-now-available-as-a-function-in-azure-stream-analytics/ .
It is more work of course but can give you more control as you control the code.
Yes,
This is actually quite easy as it is well supported by ASA.
You can call custom AzureML function from your ASA query when you create this function from the portal.
See the following tutorial on how to achieve something like this.
I have an Azure Stream Analytics job outputting to PowerBI - outputs are present and the data set is present in PowerBI - however, the graph is not live as I have to manually refresh the page to receive the updated data.
How do I go about having a live feed of data?
Cheers!
After pinning the graph to a dashboard it should refresh in real time. There are some limitations about the throughput that a PowerBI dashboard can handle and sending a large volume of events may result in latency in the view. Details in the Limitations and Best Practices topic here: https://azure.microsoft.com/en-us/documentation/articles/stream-analytics-power-bi-dashboard/