How can I monitor the following metrics for Azure SQL Database:
- Buffer Cache hit ratio.
- Page life expectancy.
- Page Splits.
- Lock waits.
- Batch requests.
- SQL compilation.
The new Azure SQL Analytics
Azure SQL Analytics is a cloud monitoring solution for monitoring
performance of Azure SQL Databases at scale across multiple elastic
pools and subscriptions. It collects and visualizes important Azure
SQL Database performance metrics with built-in intelligence for
performance troubleshooting on top.
Performance counters on SQL Azure only collect SQL Server counters of a specific database and do not show Windows performance counters (like Page Life Expectancy). For some performance counters you need to take a first snapshot, then a second snapshot, and then you should substract values of counters between snapshots to get the actual counter value.
Please use the script provided on the following article to properly collect those counters.
Collecting performance counter values from a SQL Azure database.
You are probably looking for dynamic management views. A good starting point will be
Monitoring Azure SQL Database using dynamic management views.
Regarding Buffer Cache hit, Page life etc. check this blog
SQL Server memory performance metrics – Part 4 – Buffer Cache Hit Ratio and Page Life Expectancy
Related
I have two identical resource groups for dev and qa.
They have the same services and same configurations.
At some point, I found that appinsight in QA contains a lot of performance counters which are produced by the app service plan.
I've tried to compare their configuration, but they look similar.
What can cause such different behaviour?
According to Document it says,
At the set sample interval, Azure Monitor gathers data from all deployed performance counters on all agents. For the time period defined by your log analytics workspace, the raw data is accessible in all log query views and has not been aggregated.
According to Document it says,
Performance counters show how the system resources are performing. Performance counter data ingestion depends on your environment size and usage. In most cases, performance counters should make up 80 to 99% of your data ingestion for Azure Monitor for Azure Virtual Desktop.
References:
https://learn.microsoft.com/en-us/dotnet/framework/debug-trace-profile/performance-counters
We are storing our Windows/Linux VM metrics and logs into Azure diagnostics storage account for long term retention. We keep this data in Log Analytics as well but being cost conscious we keep only the minimal essential set and for 1 month. However it seems there is no way to efficiently query the Table storage data when we need it - e.g. checking historical cpu usage for a particular machine over a specific period in the past, or checking the logs captured during that period. The partition key and row key is highly convoluted with some very basic help available for the WAD tables schema while none exist for LinuxsyslogVer2v0 table schema. I was curious if anyone else using the diagnostic logs table storage for any querying/reporting? If so how do you query these for a specific host and time period? I can do a querying using non primary/row key but besides being time consuming it will cost a hell eventually considering that will be a table scan. Really appreciate any advice.
You should consider using Azure Data Explorer (ADX) for your long-term storage solution. It allows for KQL queries on your long-term data and is the preferred method for keeping log/security data past the default for services like LogA and Sentinel.
The pricing page for ADX can be a bit confusing and there is a website to help you estimate costs here: https://dataexplorer.azure.com/AzureDataExplorerCostEstimator.html
By default, logs ingested into Azure Sentinel are stored in Azure Monitor Log Analytics. This article explains how to reduce retention costs in Azure Sentinel by sending them to Azure Data Explorer for long-term retention.
Storing logs in Azure Data Explorer reduces costs while retains your ability to query your data, and is especially useful as your data grows. For example, while security data may lose value over time, you may be required to retain logs for regulatory requirements or to run periodic investigations on older data.
https://learn.microsoft.com/en-us/azure/sentinel/store-logs-in-azure-data-explorer?tabs=adx-event-hub
We have legacy applications that currently write out various run time metrics (SQL calls run time, api / http request run times etc) to local SQL DB.
format:( source, event, data, executionduration)
We are moving away from storing those in local SQL DB, and are now publishing those same metrics to azure event hub.
Looking for a good place to store those metrics for the purpose of monitoring the health of the application. Simple solution would be to store in some DB and build custom application to visualize the data in custom ways.
We are also considering using Azure Monitor for this purpose via data collector API (https://learn.microsoft.com/en-us/azure/azure-monitor/platform/data-collector-api)
QUESTION: Are there any issues with azure monitor that would prevent us from achieving this type of health monitoring?
Details
each event is small (few hundred characters)
expecting ~ 10 million events per day
retention of 1-2 days is enough
ability to aggregate old events per source per event is important (to have historical run time information)
Thank you
You can do some simple graphs and with the Log Analytics query language, you can do just about any form of data analytics you need.
Here's a pretty good article on Monitor Visualizations.
learn.microsoft.com/en-us/azure/azure-monitor/log-query/charts
What is the maintenance required from an organization when deploying an Azure SQL Database in the long term?
My current organization is hoping to do as little database management as possible, and have looked for products that fully manage our databases without much intervention needed from our end. Some products that are being considered includes Snowflake (for their automated partitioning of tables) and Domo (for their data warehousing, connectors, and BI tool offerings).
I'm leaning towards using Azure SQL DB for multiple reasons (products offered, transparent pricing, integration ease, available documentation, SSO, etc.), but want to first understand the skills needed and ease in maintaining it in the long run.
Will we have to manually rebuild indexes and partition out tables as we scale up? Or is Azure intelligent enough that it'll do most of the heavy lifting of performance optimization itself?
Does Azure or other vendors provide services to optimize a DB?
Sorry for the vague prompts, but any additional considerations in choosing DB vendors would be great. Thanks!
Actually for your questions, you should know what is Azure SQL database and it's capabilities.
I'm leaning towards using Azure SQL DB for multiple reasons (products offered, transparent pricing, integration ease, available documentation, SSO, etc.), but want to first understand the skills needed and ease in maintaining it in the long run.
This document What is Azure SQL Database service introduced almost all message you want to know. SQL Database is a general-purpose relational database managed service in Microsoft Azure that supports structures such as relational data, JSON, spatial, and XML. SQL Database delivers dynamically scalable performance within two different purchasing models: a vCore-based purchasing model and a DTU-based purchasing model. SQL Database also provides options such as columnstore indexes for extreme analytic analysis and reporting, and in-memory OLTP for extreme transactional processing. Microsoft handles all patching and updating of the SQL code base seamlessly and abstracts away all management of the underlying infrastructure.
Will we have to manually rebuild indexes and partition out tables as we scale up? Or is Azure intelligent enough that it'll do most of the heavy lifting of performance optimization itself?
No, you don't. Scalability is one of the most important characteristics of PaaS that enables you to dynamically add more resources to your service when needed. Azure SQL Database enables you to easily change resources (CPU power, memory, IO throughput, and storage) allocated to your databases.
You can mitigate performance issues due to increased usage of your application that cannot be fixed using indexing or query rewrite methods. Adding more resources enables you to quickly react when your database hits the current resource limits and needs more power to handle the incoming workload. Azure SQL Database also enables you to scale-down the resources when they are not needed to lower the cost.
For more details, please reference: Scale Up/Down.
Does Azure or other vendors provide services to optimize a DB?
As Woblli said, Azure SQL database provides the Azure SQL database Monitoring and tuning for you.
As a complement, you also can use Azure SQL Database Automatic tuning to help you optimize the database automatically.
Hope this helps.
Azure SQL DB offers the services you're asking.
You can enable automatic tuning, which will create and drop indexes based on performance gains. Force good query plans again based on performance. It will roll back changes if the specific change has decreased the overall database performance level.
It will not partition or shard your database for you however.
Official documentation:
https://learn.microsoft.com/en-us/azure/sql-database/sql-database-automatic-tuning
As far as I know the key points to migrate an existing database to SQL Azure are:
Tables has to contain a clustered
index. This is mandatory.
Schema and data migration should be
done through data sync, bulk copy,
or the SQL Azure migration
wizard, but not with the restore option in SSMS.
The .NET code should handle the
transient conditions related with
SQL Azure.
The creation of logins is in the
master database.
Some TSQL features may be not
supported.
And I think that's all, am I right? Am I missing any other consideration before starting a migration?
Kind regards.
Update 2015-08-06
The Web and Business editions are no longer available, they are replaced by Basic, Standard and Premium Tiers.
.CLR Stored Procedure Support is now available
New: SQL Server support for Linked Server and Distributed Queries against Windows Azure SQL Database, more info.
Additional considerations:
Basic tier allows 2 GB
Standard tier allows 250 GB
Premium tier allow 500 GB
The following features are NOT supported:
Distributed Transactions, see feature request on UserVoice
SQL Service broker, see feature request on UserVoice
I'd add in bandwidth considerations (for initial population and on-going bandwidth). This has cost and performance considerations.
Another potential consideration is any long running processes or large transactions that could be subject to SQL Azure's rather cryptic throttling techniques.
Another key area to point out are SQL Jobs. Since SQL Agent is not running, SQL Jobs are not supported.
One way to migrate these jobs are to refactor so that a worker role can kick off these tasks. The content of the job might be moved into a stored procedure to reduce re-architecture. The worker role could then be designed to wake up and run at the appropriate time and kick off the stored procedure.