I have implemented an Azure alert that should fire when a Application Insights metric is greater than zero. The metric is the number of items in a Azure poison queue. The metric is calculated by a Azure Function described in this article: https://www.scaling-to-the-sky.com/2018/03/08/poison-queue-monitoring-with-azure-functions/?unapproved=160&moderation-hash=072116753136d2008f5e63a856d8e4b0#comment-160.
The alert has only fired once despite the condition being met on several occasions. I don't know why it doesn't fire. I have noticed that on the one alert that was fired the monitor condition has never changed from "Fired" to "Resolved". Maybe that is the reason why no need new alerts are fired? If that is the case then how do I change the state of the alert monitor condition?
A metric alert that's in a "Fired" state would not trigger again until it's resolved. This is done to reduce noise. Resolution happens automatically after 3 healthy evaluations of your condition (evaluations where the condition isn't breached), and there's no way to manually change the monitor condition to "Resolved".
Can you please confirm if you are sending a metric value on every evaluation of the poison queue, even if the value is 0?
Metric alerts are stateful by default, so other alerts aren't fired if there's already a fired alert on a specific time series.
To make a specific metric alert rule stateless and get alerted on every evaluation1 in which the alert condition is met, use one of these options:
If you create the alert rule programmatically, for example, via Azure Resource Manager, PowerShell, REST, or the Azure CLI, set the autoMitigate property to False.
If you create the alert rule via the Azure portal, clear the Automatically resolve alerts option under the Alert rule details section.
Check out the details here:
Make metric alerts occur every time my condition is met
Related
I have a stateful metric alert which triggers a logic app when the maximum count of a dead-letter queue/topic is greater than 5
(this is a standard metric alert, not a custom one)
The alert is configured with:
Evaluation frequency 5min
Aggregation timeframe 5min
Alert overview:
Problem
Azure monitor executes the action group and triggers the logic app even when the condition is NOT met. Navigating to the Logic app and inspecting the http trigger's payload:
I've even seen values of metricValue: 0 why would that be the case?
I want to trigger the alerts that i have configured for a system manually, we don't have access to the infrastructural design of the system.
Azure Monitor only allows you to manually create alerts that will notify you when a metric crosses a predefined threshold.
you’ll be able to add a condition to trigger the alert and can set the threshold you’d like to alert on but we cannot trigger an azure monitoring alert manually.
I am trying to schedule my azure alerts , which runs for every five minutes by query and if condition met, it triggers an email via action group.
what my requirement is to stop email alerts on weekends, I have tried azure action processing rules but it is not working at all.
where am I wrong couldn't figure it out, can anyone suggest that which approach would be better and how to do that and does action processing rules works?
To suppress notifications for your alerts would be to use the Alert Processing Rules feature, which is currently in preview.
The Alert processing rules that modify the fired alerts themselves as they are being fired. You can use alert processing rules to add action groups or remove (suppress) action groups from your fired alerts. In your requirement you could use Suppress.
Suppress approach has some limitations:
You could disable the relevant alert rule at the beginning of the
maintenance window. Once the maintenance is over, you can then
re-enable the alert rule. However, this approach is only practical if
the scope of the alert rule is exactly the scope of the resources
under maintenance. For example, a single alert rule might cover
multiple resources, but only one of those resources is going through
maintenance. So, if you disable the alert rule, you will miss valid
alerts on the remaining resources covered by that rule.
You may have many alert rules that cover the resource. Updating all
of them is time consuming and error prone.
You might have some alerts that are not created by an alert rule at
all. In all these cases, an alert processing rule provides an easy
way to achieve the notification suppression goal.
You can add your Suppression alert rule to disable the notification. I have disabled the Email Notification on weekends which follows
Refer Document for more information
Is there a way to monitor the number of times a runbook has been called and then report on it (send email, text)? When I try to create an alert rule I only see an option for activity log not metrics. The runbook is getting called from event grid via webhook.
You can use automation runbooks with three alert types:
Classic metric alerts - Sends a notification when any platform-level metric meets a specific condition. For example, when the value for CPU % on a VM is greater than 90 for the past 5 minutes.
Activity log alerts
Near real-time metric alerts - Sends a notification faster than metric alerts when one or more platform-level metrics meet specified conditions. For example, when the value for CPU % on a VM is greater than 90, and the value for Network In is greater than 500 MB for the past 5 minutes.
When an alert calls a runbook, the actual call is an HTTP POST request to the webhook. The body of the POST request contains a JSON-formated object that has useful properties that are related to the alert.
This Microsoft documentation link might help for metrics alerts for runbook :
https://learn.microsoft.com/en-us/azure/automation/automation-create-alert-triggered-runbook
You can send your Azure Automation runbook status data to Log Analytics. From there, you can alert on the different states. This documentation should help you with this process: https://azure.microsoft.com/en-us/updates/send-your-runbook-job-status-and-job-streams-from-automation-to-log-analytics-oms/
There are multiple answers to this question although none of them are a perfect solution. What I ended up doing was putting a logic app in front of the runbook that then calls the runbook. This allows me to alert on the metrics of my logic app.
We have just added support for Monitoring runbooks and having Alerts on it. Please go to the Alerts experience in Azure Monitor where you should be able to choose an Automation Account and then a Runbook as a dimension and then perform actions based on number of jobs of the Runbook etc.
I need to set an alert on custom metric in Application Insights. For testing, I sent custom metric from a console app and then in Azure portal, set the alert on that custom metric. but when we will move the app to production, how we can make it work. I assume , during installation, we will need to set the alerts but I will be sending the custom metric through my application, so how I will set alert on that metric in the beginning which does not exists yet.
There's a current limitation that you have to send a value for the metric at least once, to cause it to be a "known" / "valid" metric name to create an alert for.
John is correct. This is a desired behavior aimed at preventing customers from making an error when specifying the name of a metric and thus configuring and relying on an alert that will never fire.