In order to write sensor data from an IoT device to a SQL database in the cloud I use an Azure Streaming Analytics job. The SA job has an IoT Hub input and a SQL database output. The query is trivial; it just sends all data through).
According to the MS price calculator, the cheapest way of accomplishing this (in western Europe) is around 75 euros per month (see screenshot).
Actually, only 1 message per minute is send through the hub and the price is fixed per month (regardless of the amount of messages). I am surprised by the price for such a trivial task on small data. Would there be a cheaper alternative for such low capacity needs? Perhaps an Azure function?
If you are not processing the data real-time then SA is not needed, you could just use an Event Hub to ingest your sensor data and forward it on. There are several options to move data from the Event Hub to SQL. As you mentioned in your question, you could use an Azure Function or if you want a no-code solution, you could us a Logic App.
https://learn.microsoft.com/en-us/azure/connectors/connectors-create-api-azure-event-hubs
https://learn.microsoft.com/en-us/azure/connectors/connectors-create-api-sqlazure
In addition to Ken's answer, the "cold path" can be your solution, when the telemetry data are stored in the blob storage by Azure IoT Hub every 720 seconds (such as a maximum batch frequency).
Using the Azure Event Grid on the blob storage, it will trigger an EventGridTrigger subscriber when we can handle starting a streaming process for this batch (or for a group of batches within an one hour). After this batch process is done, the ASA job can be stopped.
Note, that the ASA job is billed based on the active processing time (that's the time between the Start/Stop) which your cost using an ASA job can be significantly dropped down.
Related
I have a device which sends data in the form of Hexadecimal value based on an algorithm to IoT Hub. I want to parse that data into a json string to store it in Cosmos DB. Is there any way to achieve this?
I would like to add to Roman's comment on your question, he's right to offer this link to Stream Analytics. It will get the job done. Depending on how many devices you have and how often you are receiving telemetry, you might want to consider using Azure Functions instead. See this sample on how to integrate Azure Functions between IoT Hub and CosmosDB.
The reason I offer this extra solution is that a Stream Analytics Job will cost you a fixed price per hour per streaming unit, while a Function is paid by consumption. Because the conversion from hexadecimal is a fairly small Function, you might even use it for free whereas a Stream Analytics Job in West Europe will cost at least 74 Euros.
I am looking at pricing information for Azure Stream Analytics using the Azure Pricing Calculator: https://azure.microsoft.com/en-us/pricing/calculator/#stream-analytics378d63b3-303d-4548-afd0-bac5d4249611
I says that it cost 80 USD for 1 month (730 hours).
Is this the idle price?
If i don't send any data to Steam Analytics I will be charged 80 USD a month for having the service deployed. Correct?
Then op top of that I pay for my actual usage in form of streaming units, where a streaming unit, according to the FAQ, is a blend of compute, memory, and throughput.
As a cloud service, Stream Analytics is optimized for cost. There are no upfront costs involved - you only pay for the streaming units you consume, and the amount of data processed. There is no commitment or cluster provisioning required, and you can scale the job up or down based on your business needs.
While creating stream Analytics Job, if you created a Stream Analytics job with streaming units = 1, it will be billed $0.11/hour.
If you created a Stream Analytics job with streaming units with N, it will be billed $0.11 * N/hour.
Hope this helps. If you have any further query do let us know.
NO. You will pay only for what you use. For example if you configure 1 streaming unit and used 10 hours per month (10*0.11). If you configure 2 streaming unit and used 20 hours per month (20*0.11*2)
1.If i don't send any data to Steam Analytics I will be charged 80 USD a
month for having the service deployed.
Quick answer is yes.Based on this document:
As a cloud service, Stream Analytics is optimized for cost. There are
no upfront costs involved - you only pay for the streaming units you
consume, and the amount of data processed. There is no commitment or
cluster provisioning required, and you can scale the job up or down
based on your business needs.
SU is basis of ASA job. Choosing the number of required SUs for a particular job depends on the partition configuration for the inputs and the query that's defined within the job.More details,please refer to this link.
2.Then op top of that I pay for my actual usage in form of streaming
units, where a streaming unit, according to the FAQ, is a blend of
compute, memory, and throughput.
If your data is not updated in real time, or very small,maybe ASA is not your Best choice. You could follow some suggestions in this thread:Azure Stream Analytics job expensive for small data?
In the context of Azure IoT hub, when would one use Stream Analytics over Time Series Insights?
The product pages and documentation for both indicates they are heavily geared for IoT/data applications. However, I'm not clear on the differences.
The use case I have is both real time monitoring as well as ETL analysis. Could (or even should?) the two be used together?
One immediate difference I can see is that Time Series Insights stores the data whereas Stream Analytics (I think) would need the developer to integrate storage.
In short, stream analytics is about transforming, filtering and aggregation of data and time series insight is about visualising (stored) data.
Data passed through stream analytics is typically forwarded to resources like power bi (for realtime monitoring) or storage like a database for later analysis or processing.
One immediate difference I can see is that Time Series Insights stores the data whereas Stream Analytics (I think) would need the developer to integrate storage.
This is a correct statement. TSI is a data store, but its purpose is to create an environment to (visually) analyze that data. ASA cannot be used to analyze data on its own.
You could use ASA to transform the data and have the data send to Event Hub. That same Event Hub can then be used as a data source for TSI.
I have just started learning Azure IoT and it's quite interesting. I am confuse about does IoT Hub stores data somewhere?
i.e. Suppose i am passing room Temperature to IoT hub and want to store it in database for further use. How it's possible?
I am clear on how device-to-cloud and cloud-to-device works with IoT hub.
IoT Hub exposes device to cloud messages through an event hubs endpoint. Event Hubs has a retention time expressed in days. It's a stream of data that the reading client could re-read more time because the cursor is on client side (not on server side like queues and topics). With IoT Hub the related retention time is 1 day by default but you can change it.
If you want to store received messages from device you need to have a client reading on the Event Hubs exposed endpoint (for example with an Event Processor Host) that has the business logic to process the messages and store them into a database for example.
Of course you could use another decoupling layer so that the client reads from event hubs and store messages into queues. Then you have another client that at its own pace reads from queues and store into database. In this way you have a fast path reading event hubs.
This is pretty much the use case for all IoT scenarios.
Step 1: High scale data ingestion via Event Hub.
Step 2: Create and use a stream processing engine (Stream Analytics or HDInsight /Storm). You can run conditions (SQL like queries) to filter and store appropriate data in either cold or hot store for further analytics.
Step 3: Storage for cold-path analytics can be Azure BLOB. Stream Analytics can directly be configured to write the Data into it. Cold can contain all other data that doesn't require querying and will be cheap.
Step 4: Processing for hot-path analytics. This is data that is more regularly queries for. Or data where real time analytics needs to be carried on. Like in your case checking for Temperature values going beyond a threshold! needs an urgent trigger!
Let me know if you face any challenges while configuring the Stream analytics job! :)
If you take a look at the IoT Suite remote monitoring preconfigured solution (https://azure.microsoft.com/documentation/articles/iot-suite-remote-monitoring-sample-walkthrough/) you'll see that it persists telemetry in blob storage and maintains device status information in DocumentDb. This preconfigured solution gives you a working illustration of the points made in the previous answers.
We have an IoT service running on Azure which produces a lot of events. We need to build a new feature which allows our end-user to configure alerts based on system events. It allows the user to pick an event and configure an action (e-mail, webhook, etc) to be executed when such an event occurs. We're evaluating Azure Eventhub and possibly Azure Stream Analtyics as candidates for the job.
The problem we face is: We think we will get a lot of stream analytics jobs running. When for example we have 3000 customers each configuring 3 alerts we need to run 9000 stream analtyics jobs which select specific events from the eventhub pushing it in a queue which does the alert processing. This will not only be a tough maintenance job, but I think it not a really cost effective solution.
Any thoughts on this or better solutions?
I am assuming events from multiple customers go to a fixed small set of event hubs and actions go to a fixed small set of queues.
You can design it such that a single Azure stream analytics job handles processing for multiple customers. Reference Data (lets call it CustomerToAlertLookup) can be used to decide which customer has configured which alert, and event stream can be joined with CustomerToAlertLookup to decide if there should be an alert or not.
Number of Azure stream analytics jobs required would be a factor of number of outputs and manageability preferences.
I can attempt to write a sample query if you add a hypothetical but more concrete scenario.