Usecase: Trigger Azure Function only for predefined Azure activity logs.
I tried to configure Azure Activity logs and Export to Event Hub, but it won't allow Filter set on it. As per Azure document, the filter settings do not have an impact on export settings.
My usecase is to trigger an Azure Function only for a specific set of activity logs (say VM, VNet, NSG Create/Delete/Modify). What other Azure services can I use to accomplish this?
One option, but with some constraints, is to create Alerts at Resource Group level or even for specific resources. Alerts provide some flexibility in filtering specific events for which you would want to trigger an Action, say an Azure Func in your case.
I was thinking Azure Logic Apps would do this as well. However, to my surprise I could not find an option to add Activity Log as a trigger. Probably, it would come in the future. As Azure is updated quite frequently, keep checking every now and then to see if you get any new options to do this.
Related
There's an awful lot of disjointed documentation on monitoring network/resources in Azure. What I'm looking for is which pieces are needed to get information from VMs, NVA firewalls, azure load balancers, and other network resources and network connectivity into a single pain of glass in Azure. Only concerned about Azure, not on-prem for now.
I've come across azure monitor, log analytics work spaces, event hub, vm extensions, network watcher, insights, etc...but I'm not sure which are required and which are not. One doc leads to the next and I end up with 30 tabs open. I'll also need to be able to push logs to other security devices such as a SIEM.
Does anyone know of a deployment guide that wraps this all up in a more logical fashion? Does anyone have any feedback on which pieces from azure (not 3rd parties) are required at a minimum to accomplish a single pane of glass to view my Azure environment holistically?
General overview of observability in Azure
Likely, the thing you're looking for is Azure Monitor. It's an umbrella term for everything observability related inside Azure.
To store Metrics and Logs you need Log Analytics: it can query data with kusto query language, visualize results, define Alerts on queries.
Alerts is quite a complex beast, as it is spread across the entire cloud. Two types that I use the most:
log-analytics alert (which I mentioned above)
Alerts tab, which is available at every Azure component view. for example, open resource group, and scroll down to Monitoring section
Each component also has a subset of built-in metrics. Likely, you noticed that many azure components on the Overview view display some charts. For example, Azure Storage Account displays Total egress, Total ingress, and other line-charts. When you click on these charts you can customize them. These metrics and charts are free to use.
Microsoft also has all-in-one observability solution for Azure Functions and Web Apps: Application Insights
Dashboards allows to join multiple charts into a single view and share it with others.
If you care about security, Azure proposes Azure Security Center
Deployment/management strategy
I suggest to start with:
Create Log Analytics Workspace, which is the storage for metrics and logs. The azure docs article explains how to design it: how many instances to use, how to rate-limit ingestion (it might be expensive if goes out of control), how to access it and so on.
To get Azure components logs, look for Diagnostic Settings tab at a component page at Azure portal, but not all components has it (sic!). I suggest
sending the most critical data to Log Analytics workspace to store them in a queryable format for 30 days (it's in free tier). This is needed for investigating current issues with your infrastructure
if you might need logs later than 30 days - send them to Storage Account
you mentioned SIEM integration - route required events to Event Hub and then process the stream according to your requirements
So, if you need long-term storage - you need to create Azure Storage Account.
If you need real-time analysis - you need to build a pipeline based on Azure Event Hub.
If you have Azure Functions and Web Apps - add Application Insights. According to my experience, I would suggest starting with a separate instance per each Azure Function resource or Service.
Create Alerts for each component separately. If you do it through UI - open component page at the portal and look for Alerts tab there. If you're automating the process (please do so as soon as possible), do not expect easy trip: I used ARM templates and terraform - in both cases, there are dozens of barely documented features.
Join related components core-metrics into Dashboards and share it with the team. This guide is a good starting point. Note, when you share the dashboard, it's also persisted as an azure resource in the subscription.
I want to have a control in Azure regarding new and deleted items
I need a query to know "who" and "when" a resource is created or deleted in Azure
Is this possible? How can I do this query?
I need a query to know "who" and "when" a resource is created or
deleted in Azure
Is this possible? How can I do this query?
Whenever a resource is created or deleted, information about that operation is stored in Azure Activity Logs. You should be able to find the information by querying that.
Another alternative would be to make use of Azure Event Grid and subscribe to Subscription Events. You can subscribe to Microsoft.Resources.ResourceWriteSuccess (for creation/updation of resources) and Microsoft.Resources.ResourceDeleteSuccess (for resource deletion) events and take action on these events in near real time.
Within the Azure Portal, you can view these types of events from the past 90 days in the Activity Log blade.
For access to events occurring more than 90 days in the past, you need to pre-emptively set up log archival as detailed in the Export the Azure Activity Log article.
If you are planning to use the export Activity Log feature, please make sure you use the new diagnostic setting feature on Azure subscription to export Activity Logs. This feature offers multiple improvements over the old features such as Logprofiles or the Activity Log solution (Log Analytics).
https://learn.microsoft.com/en-us/azure/azure-monitor/platform/activity-log-collect
https://learn.microsoft.com/en-us/azure/azure-monitor/platform/diagnostic-settings-template
Is there a way to trigger a Logic App on a deletion of a record in an Azure SQL table?
I've checked the SQL Connector and there is only When an item is created and When an item is modified, which gives me the C and U in CRUD, but sadly there isn't an out-of-the-box trigger for the D.
I can think of some awful way of polling to get record deletions, but I'm hoping that there is a cleaner solution that some bright person has come up with, however I've had no joy with the Google searching.
I would look at the Azure Event Grid. Azure Event Grid allows you to easily build applications with event-based architectures. First, select the Azure resource you would like to subscribe to, and then give the event handler or WebHook endpoint to send the event to. Event Grid has built-in support for events coming from Azure services, like storage blobs and resource groups. Event Grid also has support for your own events, using custom topics.
I would suggest monitoring the resource group and triggering it off the deletion from the RG. There is a tutorial that shows this same concept with a VM but you should be able to modify it to meet your needs with an Azure SQL DB.
https://learn.microsoft.com/en-us/azure/event-grid/monitor-virtual-machine-changes-event-grid-logic-app
I added an on delete trigger which adds the id of the deleted record to a secondary table. I have the logic app look for modifications on the secondary table.
I found a way to query the AzureActivity Log in Azure to get a result set of changes that were made to Service Bus Topics like add, rename, delete. The result of changes includes info like TimeGenerated, HttpRequest, ResourceId. But this result set of changes is only for structural changes.
I'd like to configure my Azure instance to log the actual messages that are added to the topic, not just structural changes to a topic definition. The result set should include the same fields of TimeGenerated, HttpRequest, ResourceId. I'm guessing that there's a way to configure Azure to do this but I'm not sure how this would be done. Can you advise how to do this?
I don't think you can query the Activity logs specific to message processing.
All the management operations like create, edit, delete are possible in Azure portal, so they are capturing those events and allow us to query them.
There is no support for any kind of message processing in Azure portal and it is possible only through APIs and Libraries. So, they cannot track something which happened outside the portal.
Actually I am working on an event based notification which will call a webhook/send an email in case a new resource gets added to my subscription. I could not find any Azure native service which will send me alerts/notification in case a new resource gets added to my subscription, but using powershell we can keep on polling on the resource list and workaround it, but if there is any native Azure service or function which can help me out, please inform.
You can use the azure activity log alerts to create a web hook/sms/email notification. This allows you to create a direct avenue straight from the Azure tooling. You can also use a variety of other methods to monitor your azure subscription including Log Analytics (event based logging, alerting and analytics) and App Insights (application logging and alerting)