How can this be done? AFAIK you can stream to Event Hub only via "Diagnostic settings" / Diagnostics extension, but custom logs are AFAIK only available for Log Analytics Agent. I am a bit confused by the many Azure VM agents. In AWS I would use the Firehose Agent for streaming custom logs to Kinesis, for instance. Very simple.
You can use Diagnostic Logging to not just forward to Log Analytics but also store to Azure Storage (BLOB) or to Event Hub from Event Hub you can forward to whatever Azure Service which supports this method or from Event Hub to Azure Data Explorer.
ADX is a service specifically made for Logs\Telemetry or other kinds of data.
An example below for Logs to Event Hub:
https://learn.microsoft.com/en-us/azure/active-directory/reports-monitoring/tutorial-azure-monitor-stream-logs-to-event-hub
How to use Azure Data Explorer to ingest Logs:
https://learn.microsoft.com/en-us/azure/data-explorer/ingest-data-no-code?tabs=diagnostic-metrics
Related
I'm using an Azure IoT Hub. I'm still in the development phase. It used to work fine, but now the hub is disconnecting the devices almost immedially after they connect. Where can I see some logs or info about why the hub is disconnecting? And if I have to activate some services, which ones?
You may need to turn on diagnostics for IoT Hub for logging the device connection events and errors. Once the logs and alerts are ON for connected devices, you will get alerts, error logs when errors occur. The troubleshooting link to begin with can be https://learn.microsoft.com/en-us/azure/iot-hub/iot-hub-troubleshoot-connectivity which captures details about how to enable diagnostics, alerts and other possible troubleshooting methods. This section 'Resolve connectivity errors' has description on how to look for common issues when you receive an alert and this seems to have dependency on Azure monitor logs to be enabled. It also furnishes information in terms of problem resolution guides for the most common errors.
There are couple of services integrated with IoT Hub like Azure Monitor and Azure Resource Health that help to provide you with the data required for keeping your IoT solution running in healthy state. Azure Resource Health helps to monitor whether your IoT hub is up and running. Here is a related link on iot hub health monitor and diagnose problems that can be a additional reference for you.
Azure Monitor stores log data in a Log Analytics workspace, which is an Azure resource and a container where data is collected, aggregated, and serves as an administrative boundary as conveyed in Azure monitor logs. Data in Azure Monitor Logs is retrieved using a log query written with the Kusto query language, which allows you to quickly retrieve, consolidate, and analyze collected data.
We are currently investigating methods in getting our Security log data out of our Azure VM's and into our SIEM for analysis.
Currently I have been able to get the logs from the VM to log analytics work-space but I'm no sure how to get them from the log analytics workspace to the eventhub to then pull down the events.
Has anyone faced a similar challenge before / how did you overcome this challenge?
I'm currently pull the data into a Log analytics workspace
Welcome to Stackoverflow!
Azure diagnostic logs can be streamed in near real time to any application using the built-in “Export to Event Hubs” option in the Portal, or by enabling the Event Hub Authorization Rule ID in a diagnostic setting via the Azure PowerShell Cmdlets or Azure CLI.
What you can do with diagnostics logs and Event Hubs:
Here are just a few ways you might use the streaming capability for Diagnostic Logs:
Stream logs to 3rd party logging and telemetry systems – You can stream all of your diagnostic logs to a single event hub to pipe log data to a third-party SIEM or log analytics tool.
View service health by streaming “hot path” data to Power BI – Using Event Hubs, Stream Analytics, and Power BI, you can easily transform your diagnostics data in to near real-time insights on your Azure services.
Build a custom telemetry and logging platform – If you already have a custom-built telemetry platform or are just thinking about building one, the highly scalable publish-subscribe nature of Event Hubs allows you to flexibly ingest diagnostic logs.
After data is displayed in the event hub, you can access and read the data in two ways:
Configure a supported SIEM tool. To read data from the event hub, most tools require the event hub connection string and certain permissions to your Azure subscription. Third-party tools with Azure Monitor integration included.
For more details, refer "Stream Azure Diagnostic Logs to an event hub" and "How to integrate Azure Monitor with SIEM tools".
Hope this helps.
You can’t pull the VM data from log analytics to an event hub, you can use windows/Linux diagnostic extensions to route data to an event hub.
Stream Azure monitoring data to an event hub for consumption by an external tool
I would like to build a trigger like thing that can monitor my Azure File Storage for new coming files. If there are new files then create a VM.
What should I look at? Monitor + Alert or Logic Apps Designer?
Any one done anything similar before? Thanks!
What should I look at? Monitor + Alert or Logic Apps Designer?
I recommand that you could use Azure function+ Event Grid.
You could use the Event Grid to listen the change of the storage and implement the creating VM in the Azure function. About how to handle Event Grid events in Azure Function, please refer to this document.
Note: Currently, the following Azure services support sending events to Event Grid:
Azure Subscriptions (management operations)
Container Registry
Custom Topics
Event Hubs
IoT Hub
Media Services
Resource Groups (management operations)
Service Bus
Storage Blob
Storage General-purpose v2 (GPv2)
If Azure blob storage is possible, you could use the Azure function with blob trigger.
I'd like to trigger an Azure Function when a file is added to an Azure File share. I've not seen any documentation on this. Is it supported?
How would I get that done?
Event grid does not support File share but there is an open GitHub issue which says it is likely to be implemented when Azure Functions V2 goes GA (General Availability).
Currently, the following Azure services support sending events to Event Grid:
Azure Subscriptions (management operations)
Custom Topics
Event Hubs
IoT Hub
Media Services
Resource Groups (management operations)
Service Bus
Storage Blob
Storage General-purpose v2 (GPv2)
Reference Docs: https://learn.microsoft.com/en-us/azure/event-grid/overview
I have an IoT Hub with custom endpoints which sink data into service bus queues. Since i implemented custom routing through custom endpoints in IoT Hub & Service bus queues. When i try to create Azure Stream Analytics (ASA) job, the input panel in azure portal does not list custom endpoints but only consumer groups which work only for default messaging endpoint.
Is there a way to let ASA to pick messages off service bus queues or custom endpoints in IoT Hub so i do not need to modify custom routing i already have.
Thanks and looking forward to your thoughts.
Yes, Azure Stream Analytics (ASA) can access these endpoints. However they are not exposed when you select IoT Hub as input in ASA.
If you want to access these endpoints in ASA, you will need to point your ASA input to the Service Bus or Event Hub hosting these routes.
Let me know if it works for you.
Thanks,
JS