I am building an application in Azure, and I really like the application Insights "Application Map". I would like to integrate IoTHub to the map, so I can check the entire stack from device to webinterface in one central place.
What I have tried:
I tried creating a diagnostic setting directly in the IoTHub.
For destination I used the same Log Analytics workspace as my application insights is using.
Results:
I can see logs using the "Logs" tab under the IoTHub, but nothing in the application insights.
The actual question:
How do I connect the IoTHub to the application insights? If this is not possible, what do you normally do to check if everything is as it should be inside the IoTHub?
Application Insights is a feature of Azure Monitor that provides extensible application performance management (APM) and monitoring for live web apps.
What metrics do you plan to track through Application Insights?
When you have critical applications and business processes relying on Azure resources, you want to monitor those resources for their availability, performance, and operation. You can use the features of Azure Monitor to analyze, monitor data generated by Azure IoT Hub and setup alerts.
Azure IoT Hub creates monitoring data using Azure Monitor, which is a full stack monitoring service in Azure that provides a complete set of features to monitor your Azure resources in addition to resources in other clouds and on-premises.
Start with the article Monitoring Azure resources with Azure Monitor, which describes the following concepts:
What is Azure Monitor?
Costs associated with monitoring
Monitoring data collected in Azure
Configuring data collection
Standard tools in Azure for analyzing and alerting on monitoring data
Please see Monitoring Azure IoT Hub and Set up and use metrics and logs with an IoT hub for more details.
Related
We are working with Azure Event Hub and want to build comprehensive monitoring plate, where we could collect some metrics that are not exportable via Diagnostics settings into Azure Monitor (Log Analytics Workspace). According to this doc we can't export "Throttled Requests" or "Active Connections" for example. However we need such metrics to conduct advanced analysis and building control plate on our data platform.
How we can obtain such metrics in our storage ?
You can export the metrics using the Metrics API of the Azure Resource Manager. There are also SDKs available, e.g. for .NET.
https://learn.microsoft.com/en-us/rest/api/monitor/metrics/list?tabs=HTTP
I'm using an Azure IoT Hub. I'm still in the development phase. It used to work fine, but now the hub is disconnecting the devices almost immedially after they connect. Where can I see some logs or info about why the hub is disconnecting? And if I have to activate some services, which ones?
You may need to turn on diagnostics for IoT Hub for logging the device connection events and errors. Once the logs and alerts are ON for connected devices, you will get alerts, error logs when errors occur. The troubleshooting link to begin with can be https://learn.microsoft.com/en-us/azure/iot-hub/iot-hub-troubleshoot-connectivity which captures details about how to enable diagnostics, alerts and other possible troubleshooting methods. This section 'Resolve connectivity errors' has description on how to look for common issues when you receive an alert and this seems to have dependency on Azure monitor logs to be enabled. It also furnishes information in terms of problem resolution guides for the most common errors.
There are couple of services integrated with IoT Hub like Azure Monitor and Azure Resource Health that help to provide you with the data required for keeping your IoT solution running in healthy state. Azure Resource Health helps to monitor whether your IoT hub is up and running. Here is a related link on iot hub health monitor and diagnose problems that can be a additional reference for you.
Azure Monitor stores log data in a Log Analytics workspace, which is an Azure resource and a container where data is collected, aggregated, and serves as an administrative boundary as conveyed in Azure monitor logs. Data in Azure Monitor Logs is retrieved using a log query written with the Kusto query language, which allows you to quickly retrieve, consolidate, and analyze collected data.
I'm developing a basic Azure IoT Remote Monitoring solution with the Azure Solution Accelerator "Remote Monitoring". When I start to actually pay for services and stop using a free account, very soon the cash starts to pile up and there seem to be very many resources created behind the scenes. I'm wondering which resources I really need and which one I could throw away to save money. These are the resources that I have:
App Service plan
App Service
Network interface
Network security group
Public IP address
Virtual network
Storage account
Azure Cosmos DB account
Device Provisioning Service
Event Hubs Namespace
App Service
App Service plan
IoT Hub
Key vault
Logic app
Azure Maps Account
API Connection
Disk
Storage account (2)
Stream Analytics job
Time Series Insights environment
Time Series Insights event source
Virtual machine
CosmosDB is probably one of the more expensive resources in your list so if you can find a way to swap some other datastore for it you can save some money.
Take a look at Remote Monitoring architectural choices. The Azure IoT Remote Monitoring solution accelerator is an open-source, MIT licensed, solution accelerator. To help you speed up your IoT development process, it shows common IoT scenarios such as:
Device connectivity
Device management
Stream processing
The Remote Monitoring solution follows the recommended Azure IoT reference architecture.
This article describes the key architectural and technical choices made in each of the Remote Monitoring subsystems. However, the technical choices Microsoft made in the Remote Monitoring solution aren't the only way to implement a remote monitoring IoT solution. You should regard the technical implementation as a baseline for building a successful application and you should modify it to:
Fit the available skills and experience in your organization.
Meet your vertical application needs.
We are currently investigating methods in getting our Security log data out of our Azure VM's and into our SIEM for analysis.
Currently I have been able to get the logs from the VM to log analytics work-space but I'm no sure how to get them from the log analytics workspace to the eventhub to then pull down the events.
Has anyone faced a similar challenge before / how did you overcome this challenge?
I'm currently pull the data into a Log analytics workspace
Welcome to Stackoverflow!
Azure diagnostic logs can be streamed in near real time to any application using the built-in “Export to Event Hubs” option in the Portal, or by enabling the Event Hub Authorization Rule ID in a diagnostic setting via the Azure PowerShell Cmdlets or Azure CLI.
What you can do with diagnostics logs and Event Hubs:
Here are just a few ways you might use the streaming capability for Diagnostic Logs:
Stream logs to 3rd party logging and telemetry systems – You can stream all of your diagnostic logs to a single event hub to pipe log data to a third-party SIEM or log analytics tool.
View service health by streaming “hot path” data to Power BI – Using Event Hubs, Stream Analytics, and Power BI, you can easily transform your diagnostics data in to near real-time insights on your Azure services.
Build a custom telemetry and logging platform – If you already have a custom-built telemetry platform or are just thinking about building one, the highly scalable publish-subscribe nature of Event Hubs allows you to flexibly ingest diagnostic logs.
After data is displayed in the event hub, you can access and read the data in two ways:
Configure a supported SIEM tool. To read data from the event hub, most tools require the event hub connection string and certain permissions to your Azure subscription. Third-party tools with Azure Monitor integration included.
For more details, refer "Stream Azure Diagnostic Logs to an event hub" and "How to integrate Azure Monitor with SIEM tools".
Hope this helps.
You can’t pull the VM data from log analytics to an event hub, you can use windows/Linux diagnostic extensions to route data to an event hub.
Stream Azure monitoring data to an event hub for consumption by an external tool
I enabled the VM insights for few VMs. How can I create alerts for those metrics in Azure monitor?
How can I create alerts for those metrics in Azure monitor?
You can not create alerts for the metrics provided by VM insights. VM insight preview provides an in-depth view of VM health, performance trends, and dependencies. You can create these metrics alert for VM.
Moreover, Standard or Platform metrics are limited, you can try to make custom metrics via a variety of methods on your Azure VM to collect some custom performance indicators or business-specific metrics to provide deeper insights. Once published to Azure Monitor, you can alert on custom metrics for your Azure resources side by side the standard metrics emitted by Azure.
Ref: Send guest OS metrics to the Azure Monitor metric store using a Resource Manager template for a Windows Virtual Machine