We are looking to get alerts on specific tables in a Azure DB to initiate actions.These could be
a) calling a Azure Search indexer API so that changes get indexed in near-real time every time the datasource changes
b) push updates to users via SignalR when there is an update
I understand AzureDB has functional limitations and triggers cannot invoke the Azure function directly due to lack of CLR support.
Azure SQL Database trigger to insert audit info into Azure Table
Given the triggers in Azure cannot invoke APIs, I was told that we could get information on database updates via the auditing. However, the auditing seems to be at the blob level- not at a table level
Given the multiple changes underway, it would be quite a task and time-delay to check out for the event of interest from the blob.
While there is a line around the blob auditing being configurable(supports higher granularity object-level auditing), I couldn't find an approach to limit blob updates to certain table alone. Any pointers would be appreciated
Also, given blob auditing seems to be built for threat detection/ regulatory purposes,any issues using this for event alerts?
Additional info on granular Blob Auditing in Azure SQL DB (including limiting the audit policy to specific tables/views) can be found here:
https://msdn.microsoft.com/library/azure/mt695939.aspx
Please note that we have also created the following OMS integration app for advanced analysis of Audit logs - you can use it to push the Blob audit logs into OMS, and then create customized alerts in OMS on top of the audit logs (this is a temporary solution, until our fully integrated OMS/Log Analytics solution is available):
https://github.com/Microsoft/Azure-SQL-DB-auditing-OMS-integration
Best Regards,
Gilad Mittelman
SQL Security, Microsoft
Related
Is there a way to view the logs of when backups of Azure SQL were taken? Success, failure, etc, logs or logs of the PITR, LTR, diffs being taken?
I can see a list of our available LTRs, but I don't seem to see any log history of when these things were made, etc.
This is not SQL on Azure VMs, I'm using their fully managed Azure SQL.
You can use Azure SQL Database auditing to track database events and writes them to an audit log in your Azure storage account, or sends them to Event Hub or Log Analytics for downstream processing and analysis.
You can use SQL Database auditing to:
Retain an audit trail of selected events. You can define categories of database actions to be audited.
Report on database activity. You can use pre-configured reports and a dashboard to get started quickly with activity and event reporting.
Analyze reports. You can find suspicious events, unusual activity, and trends.
Find the detailed source document here to know more.
Set up auditing for your server tutorial here.
In SQL Azure, there are two options for getting database events in Azure portal are Auditing and Diagnostic settings.
In which scenario we will use them.
Azure SQL database Auditing is usually used for:
Retain an audit trail of selected events. You can define categories
of database actions to be audited.
Report on database activity. You can use pre-configured reports and a
dashboard to get started quickly with activity and event reporting.
Analyze reports. You can find suspicious events, unusual activity,
and trends.
Diagnostics settings:
You can use the Diagnostics settings menu for each single, pooled, or instance database in Azure portal to configure streaming of diagnostics telemetry. In addition, diagnostic telemetry can also be configured separately for database containers: elastic pools and managed instances. You can set the following destinations to stream the diagnostics telemetry: Azure Storage, Azure Event Hubs, and Azure Monitor logs.
As 4c74356b41 said, they are difference things and with different uses.
Please reference:
Azure SQL Database metrics and diagnostics logging
Get started with SQL database auditing
Hope this helps.
Just found this "When auditing is configured to a Log Analytics workspace or to an Event Hub destination via the Azure portal or PowerShell cmdlet, a Diagnostic Setting is created with "SQLSecurityAuditEvents" category enabled.".
link
For me, this means that auditing is an easier way to enable one of the features of Diagnostic Settings.
Where does data logs of Azure Pipeline v2 gets stored, I would like to retrieve data of failed pipelines for specific date.( Dont want to use azure portal to view these data). Is there any table/view holds such datalogs from database.
To my knowledge, to obtain diagnostic logs you can use Azure Monitor, Operations Management Suite (OMS), or monitor those pipelines visually.
By Azure Pipeline v2, you mean Azure Data Factory v2. Alert and Monitor data factories using Azure Monitor
Diagnostic logs:
Save them to a Storage Account for auditing or manual inspection. You can specify the retention time (in days) using the diagnostic settings.
Stream them to Event Hubs for ingestion by a third-party service or custom analytics solution such as PowerBI.
Analyze them with Log Analytics
The logs are stored on Azure Data Factory web server for 45 days. If you want to get the pipeline run and activity run metadata, you can use Azure Data Factory SDK to extract the information you need and save it somewhere you want.
Recommended approach on this for log term analysis as well as limiting access to a production data factory would be to configure logs to be sent to log analytics. Be sure to enable dedicated logging tables as this will help on the backend in terms of organizing your logs.
From there you can also set up alerts and access groups running off of log analytics queries for better monitoring.
I'm trying to set up logging for a storage resource (table specifically, though it seems like the activity log doesn't and just logs the entire Storage account).
The logging seems to log my ListKeys operations, occasional access from ApplicationInsights, but isn't logging any writes/reads I'm making to the tables themselves through either my app or the Microsoft Azure Storage Explorer. This table has been written to multiple times over the past few weeks, but yet none of that activity shows up.
Am I misinterpreting this page, which states that this activity log should track posts/deletes? Do I need any additional setup to track these operations?
Per my understanding, you could leverage Storage Analytics logging to log the operations on your storage. For the detailed operations that are logged for the corresponding storage service, you could refer to this official document.
According to your description, I have tested my operations against table storage by using REST API and Storage Explorer Tool. Here is my test result, you could refer to it.
Table Storage Analytics logging
Table Storage Metrics
As noted in this document:
As requests are logged, Storage Analytics will upload intermediate results as blocks. Periodically, Storage Analytics will commit these blocks and make them available as a blob.
In summary, please follow this tutorial to enable and configure Storage Analytics, then wait for some time and check your table storage logging.
If you are leveraging the Azure Activity log, remember that it is meant for control plane operations. So listkeys would show up there.
if you are looking for data plane operations (such as entity writes into a table), then make sure Diagnostics are turned on inside the Storage account that you are writing to.
Azure Activity Log is only for management plane records through Azure Resource Manager (ARM), specifically PUT/DELETE/POST which includes ListKeys which is an HTTP POST.
For storage analytics logging, you can use this article to see the types of data logged.
I am trying to implement azure storage metrics code in my role but i am checking if there is easy way to get azure storage metric data about my files usage. my code is stable and i do not want to change code again.
Actually if you already have Windows Azure role running, then you don't need to make any changes to your code and you still can get Windows Azure Blob storage Metrics data.
I have written a blog about it last time as Collecting Windows Azure Storage REST API level metrics data without a single line of programming, just by using tools
Please try above and see if this works for you.
Storage analytics is disabled by default, so any operations against your storage up til now has not been logged for analysis.
You may choose to enable analytics at any time, for both logging (detailed access information for every single object) and metrics (hourly rollups). Further, you may choose which specific storage service to track (blobs, tables, queues) and which operations to track (read, write, delete). Once analytics are enabled, you may access the resulting analytics data from any app (as long as you have the storage account name + key).
Persistent Systems just published a blog post on enabling storage analytics for Java apps. The same principles may be applied to a .net app (and the sdk's are very similar).
Additionally, Full Scale 180 published a sample app encapsulating storage analytics (based on REST API, as it was written before SDK v1.6 came out).