Working on a IoT telemetry project that receives humidity and weather pollution data from different sites on the field. I will then apply Machine Learning on the collected data. I'm using Event Hubs and Stream Analytics. Is there a way of pulling the data to Azure Machine Learning without the hassle of writing an application to get it from Stream Analytics and push to AML web service?
Stream Analytics has a functionality called the “Functions”. You can call any web service you’ve published using AML from within Stream Analytics and apply it within your Stream Analytics query. Check this link for a tutorial.
Example workflow in your case would be like the following;
Telemetry arrives and reaches Stream Analytics
Streaming Analytics (SA) calls the Machine Learning function to apply it on the data
SA redirects it to the output accordingly, here you can use the PowerBI to create a predictions dashboards.
Another way would be using R, and here’s a good tutorial showing that https://blogs.technet.microsoft.com/machinelearning/2015/12/10/azure-ml-now-available-as-a-function-in-azure-stream-analytics/ .
It is more work of course but can give you more control as you control the code.
Yes,
This is actually quite easy as it is well supported by ASA.
You can call custom AzureML function from your ASA query when you create this function from the portal.
See the following tutorial on how to achieve something like this.
Related
I am having some issues actually retrieving and using the data I send to the IoT Hub in Azure. When I run 'az IoT hub monitor-events --hub-name ' in CLI I can see my events, and I can also send messages to my devices in the IoT hub.
I have then tried to create a stream, to forward the messages to my SQL database, but without any luck. Do you have any suggestions on how to retrieve this data?
There are multiple ways about this. The two most common scenarios are probably using an Azure Function, or using a Stream Analytics job. I don't know what you've tried up until this point, but a Stream Analytics job is probably the easiest way to go.
Stream Analytics
This answer on SO could be what you're looking for, it also links to this tutorial that you could follow from "Create a new Azure SQL Database" onwards. It covers creating an IoT Hub input and Azure SQL output on your Stream Analytics job and using a simple query to link the two together. There is more info in the Microsoft docs here
Azure Function
While looking this one up I found this answer, which is mine, awkward. But it describes how you can go about creating an Azure Function that accepts IoT Hub messages and shoots them to your database. This option is a lot more cost-efficient (or even free, if you use the consumption plan for a Function) for a few devices.
I want to send images from my system continuously to Azure cloud and process the image on the cloud using Azure stream analytics.
Following are my requirements:
Send images from a client(my desktop) continuously to Azure.
Run my ML algorithm on the cloud on the received images.
Send the result(output image and metadata) back to the client(my system)
Which services/product of Azure would help me in doing this task in real-time??? And what the steps??
Azure Stream Analytics would most likely not be a good fit for this (not saying that you couldn't use it) since I would assume your image data would be rather large. You should rather look into Azure Machine Learning scoring web services for that. That way you can expose a web service that you can send your image to (e.g. with a POST request), score it, and get the result back.
However: there are many possible ways here and it really depends on your actual problem you are trying to solve. (I almost voted to close this question)
In the context of Azure IoT hub, when would one use Stream Analytics over Time Series Insights?
The product pages and documentation for both indicates they are heavily geared for IoT/data applications. However, I'm not clear on the differences.
The use case I have is both real time monitoring as well as ETL analysis. Could (or even should?) the two be used together?
One immediate difference I can see is that Time Series Insights stores the data whereas Stream Analytics (I think) would need the developer to integrate storage.
In short, stream analytics is about transforming, filtering and aggregation of data and time series insight is about visualising (stored) data.
Data passed through stream analytics is typically forwarded to resources like power bi (for realtime monitoring) or storage like a database for later analysis or processing.
One immediate difference I can see is that Time Series Insights stores the data whereas Stream Analytics (I think) would need the developer to integrate storage.
This is a correct statement. TSI is a data store, but its purpose is to create an environment to (visually) analyze that data. ASA cannot be used to analyze data on its own.
You could use ASA to transform the data and have the data send to Event Hub. That same Event Hub can then be used as a data source for TSI.
I'm planning to use Azure Stream Analytics on the edge and I want to be able to look-up some data from file i.e. use a reference data input.
Of course, my look-up data is static and slowly changing, but I want to be able to remotely send/deploy this file.
How should I do that? I can't find any useful info on the net about this.
Is it possible to automate this with IoT Hub and/or template deployments?
Is it possible to include this file in the docker image of the ASA job that is being deployed on the Edge run-time via IoT Hub Edge Deployment functionality?
Is there some Azure API that could be used?
Actually, this is very well supported and documented for an ASA Job running in the cloud. I want to achieve the same, but on the Edge.
To answer your questions:
At this time, the use of reference data in ASA jobs on IoT Edge is not automated, you will need to create a local reference data file first on the host based on query, and configure the data type as reference data and path as volume/path/to/.txt (linux) Disk:/path/to/.txt (windows), when creating an edge job.
We do not support storing the file in the docker image of ASA job. The only data source for reference data is local file.
There is no Azure API to be used for deploying the reference data. Once user triggers an ASA job update through IoT Edge deployment, reference data location will be updated
You can find more information on our documentation page of ASA on IoT Edge. Any customer feedback is welcomed, and we would love to explore new features that would be helpful for users.
Is there a way of automating the calls to the Azure Machine Learning Service (AML)?
I’ve created the web service from AML. Now I have to do the calls the automated way. I’m trying to build a system, that connects to a Raspberry Pi for sensor data and gets a prediction from the ML service to be saved with the data itself.
Is there something in Azure to automate this or should I do it within the application?
I'm assuming you've created the webservice from the experiment and asking about the consumption of the webservice. You can consume the webservice from anything that can do an API call to the endpoint. I don't know the exact architecture of your solution but take a look at this as it might suit your scenario.
Stream analytics on Azure has a new feature called Functions(just a heads-up, its still in preview) that can automate the usage of deployed ML services from your account.Since you are trying to gather info from IoT devices, you might use Event Hubs or IoT Hubs to get the data and process it using Stream Analytics and during the process you can use the Webservice as Function in SA to achieve on-the-go ML results.
Usage is relatively simple if you are familiar with Stream Analytics or SQL queries in general.This link shows the step by step implementation and the usage is below;
WITH subquery AS (
SELECT text, "webservicealias"(text) as result from input
)
Select text, result.[Score]
Into output
From subquery
Hope this helps!
Mert
you can also automatically schedule this using powershell command and any task scheduler
Powershell for Azure ML - https://github.com/hning86/azuremlps and its usage is described here - https://github.com/hning86/azuremlps#invoke-amlwebservicerrsendpoint
Task Scheduler for powershell - http://www.metalogix.com/help/Content%20Matrix%20Console/SharePoint%20Edition/002_HowTo/004_SharePointActions/012_SchedulingPowerShell.htm