I want to monitor some events coming from my application.
One option is to send data to Azure Event Hub and use stream analytics to do some post-processing and enter the data into cosmos db.
Another option is to store to cosmos db from application and run a periodic azure function to do the processing and store it back.
What is the right way to do it? Is there a better way to do it?
The best architecture way is to have Event Hubs to Cosmos DB. I have done the same implementations using Application -> EventHub -> ChangeFeed Azure Function -> Cosmosdb
You can read about Here.
ChangeFeed is offered by Azure Cosmos DB out of the box for this case. It works as a trigger on Cosmos DB change.
It depends on the kind of processing that you would like to do with the events ingested. If it is event at a time processing, a simple Azure Function with CosmosDB changefeed processor might be enough. If you would like to do stateful processing like windowing or event order based computation, azure Stream Analytics would be better. Stream Analytics also provides native integration to PowerBI dashboards. Same job can send the data both to CosmosDB and PowerBI. If you are going to use Azure Stream Analytics, you will have to use EventHub for event ingestion. Using EventHub for ingestion also has other benefits like being able to archive events to blob storage.
Related
I am trying to get data from my Azure IoT hub to a database, which will be then displayed on a website.
I have seen many ways to ingest data such as azure data explorer cluster and stream analytics jobs. However, the issue i have with data explorer is that i cannot get connection details for the database so i cot connect to it using code. As for stream analytics jobs, MySQL databases are not supported, and I have only have experience with MySQL.
Does anyone know a way i can ingest data from my Azure IoT hub to a MySQL database? Thanks in advance.
You will have to write some code to do this.
You can for instance create an Azure Function that uses an IoTHubTrigger or EventHubTrigger so that the Function receives the messages that are available on the IoT Hub. In the Function, you write code to make sure that the data is inserted in the appropriate tables in your database.
Find some documentation regarding this here.
I want to send Azure Diagnostics to Kusto tables.
The idea is to get logs and metrics from various Azure resources by sending them to a storage account.
I'm following both Ingest blobs into Azure Data Explorer by subscribing to Event Grid notifications and Tutorial: Ingest and query monitoring data in Azure Data Explorer,
trying to use the best of all worlds - cheap intermediate storage for logs, and using EventHub only for notifications about the new blobs.
The problem is that only part of the data is being ingested.
I'm thinking that the problem is in the append blobs which monitoring creates. When Kusto receives "Created" notification, only a part of the blob is written, and the rest of events are never ingested as the blob is appended to.
My question is, how to make this scenario work? Is it possible at all, or I should stick with sending logs to EventHub without using the blobs with Event Grid?
Append blobs do not work nicely with Event Grid ADX ingestion, as they generate multiple BlobCreated events.
If you are able to cause blob rename on update completion, that would sole the problem.
Getting error messages. Seems this is not possible ?
message is :
The JSON provided in the request body is invalid. Property 'server' value 'postgres-mydatabase123.postgres.database.azure.com' is not acceptable.
According to the documentation this is not possible as currently the support output sinks are
Azure Data Lake Store
SQL Database Blob storage
Event Hub
Power BI
Table Storage
Service Bus Queues
Service Bus Topics
Azure Cosmos DB
Azure Functions (In Preview)
Out of all of these Azure Functions or Event Hub might be interesting to you as they allow custom code to process the data. In your case that would be sending it to the PostgreSQL database.
I have a scenario where event hub gets data in every 10 seconds, which pass to the stream analytics and then which is passed to the Azure SQL Server. The technical team raised the concerns that Azure SQL is unable to handler so much of data, if data raises 2,00,00,000. then it stops to work.
Can you please guide me is it actual problem of Azure SQL, if it is then can you please suggest me the solution.
Keep in mind that 4TB is the absolute maximum size of an Azure SQL Premium instance. If you plan to store all events for your use case, then this will fill up very quickly. Consider using CosmosDb or Event Hub Capture if you really need to store the messages indefinitely and use SQL for aggregates after processing with SQL DW or ADLS.
Remeber that to optimise Event Hubs you must have a partitioning strategy to optimise the throughput. See the docs.
I am developing an application . I am pushing data to event hub .From event hub i need to use stream analytics and push data to DOcumentdb for analytics purpose. Could any one please guide me to push data from stream analytics to documentdb. I could see output to SQL Database,Blob storage,Event Hub,Power BI preview,Table storage, Service Bus Queue, Service Bus Topic only
Edit: this feature is now implemented, and DocumentDB can be used as a sink for Stream Analytics: http://blogs.msdn.com/b/streamanalytics/archive/2015/09/29/stream-analytics-updates-for-the-azure-iot-suite.aspx
Yes this effort has already been started while ago as per the feedback forum and it should be publicly available soon. Please share more feedback about your scenario over there and maybe why Azure Table or SQL output are not satisfying it.
thanks!